chloeli

qwen-2.5-32b-id-baseline

Deploy Dedicated

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Get help setting up a custom Dedicated Endpoints.

Talk with our engineer to get a quote for reserved GPU instances with discounts.

README

License: mit

Usage

Load as LoRA adapter

python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-32B-Instruct",
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, "chloeli/qwen-2.5-32b-id-baseline")
tokenizer = AutoTokenizer.from_pretrained("chloeli/qwen-2.5-32b-id-baseline")

messages = [{"role": "user", "content": "What matters most when making a difficult decision?"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Merge into base model

python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-32B-Instruct",
    torch_dtype="auto",
    device_map="cpu",
)
model = PeftModel.from_pretrained(base_model, "chloeli/qwen-2.5-32b-id-baseline")
merged_model = model.merge_and_unload()

merged_model.save_pretrained("qwen-2.5-32b-id-baseline-merged")
tokenizer = AutoTokenizer.from_pretrained("chloeli/qwen-2.5-32b-id-baseline")
tokenizer.save_pretrained("qwen-2.5-32b-id-baseline-merged")

Serve with vLLM

python
from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest

llm = LLM(
    model="Qwen/Qwen2.5-32B-Instruct",
    enable_lora=True,
    max_lora_rank=128,
)

lora_request = LoRARequest("baseline", 1, "chloeli/qwen-2.5-32b-id-baseline")
output = llm.generate("What matters most?", SamplingParams(max_tokens=512), lora_request=lora_request)