GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
sarimahsan101
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
sarimahsan101
Model Tree
Qwen/Qwen2.5-0.5B-Instruct
Input Modalities
Output Modalities
Supported Functionality
These metrics were recorded during a trial run on a single NVIDIA Tesla T4 (16GB) GPU for 1 epoch on a subset of Alpaca, Dolly, and Ultrachat:
| Metric | Before Distillation | After Distillation | Change | Status |
|---|---|---|---|---|
| Validation Perplexity | 5.0924 | 5.2620 | +0.1696 | ✗ |
| Teacher-Student KL Divergence | 2.7913 | 1.9637 | -0.8276 | ✓ |
| Hidden State Cosine Similarity | 0.0075 | 0.0054 | -0.0021 | ✗ |
To load and use this adapter, run:
python
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base_model_name = "Qwen/Qwen2.5-0.5B-Instruct"adapter_name = "sarimahsan101/Qwen2.5-0.5B-HiddenDistilled-LoRA" # Load tokenizer and base modeltokenizer = AutoTokenizer.from_pretrained(base_model_name, trust_remote_code=True)base_model = AutoModelForCausalLM.from_pretrained( base_model_name, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True) # Load LoRA adaptermodel = PeftModel.from_pretrained(base_model, adapter_name)model.eval() # Inference examplemessages = [{"role": "user", "content": "Explain gravity in one sentence."}]text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = tokenizer(text, return_tensors="pt").to(model.device) with torch.no_grad(): outputs = model.generate(**inputs, max_new_tokens=50) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))bitsandbytes)import torchfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base_model_name = "Qwen/Qwen2.5-0.5B-Instruct"adapter_name = "sarimahsan101/Qwen2.5-0.5B-HiddenDistilled-LoRA" # Load tokenizer and base modeltokenizer = AutoTokenizer.from_pretrained(base_model_name, trust_remote_code=True)base_model = AutoModelForCausalLM.from_pretrained( base_model_name, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True) # Load LoRA adaptermodel = PeftModel.from_pretrained(base_model, adapter_name)model.eval() # Inference examplemessages = [{"role": "user", "content": "Explain gravity in one sentence."}]text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = tokenizer(text, return_tensors="pt").to(model.device) with torch.no_grad(): outputs = model.generate(**inputs, max_new_tokens=50) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))