GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
sarimahsan101
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
sarimahsan101
Model Tree
Qwen/Qwen2.5-0.5B-Instruct
Input Modalities
Output Modalities
Supported Functionality
Evaluation metrics compiled on a single NVIDIA Tesla T4 (16GB) GPU for 1 epoch on a subset of Dolly, Alpaca SFT, and Ultrachat:
| Metric | Before Distillation | After Distillation | Change | Status |
|---|---|---|---|---|
| Validation Perplexity | 5.0924 | 5.2620 | +0.1696 | ✗ |
| Teacher-Student KL Divergence | 2.7913 | 1.9637 | -0.8276 | ✓ |
| Hidden State Cosine Similarity | 0.0075 | 0.0054 | -0.0021 | ✗ |
You can load this model directly using standard Hugging Face Transformers:
python
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer model_name = "sarimahsan101/Qwen2.5-0.5B-HiddenDistilled" # Load tokenizer and merged modeltokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)model = AutoModelForCausalLM.from_pretrained( model_name, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True)model.eval() # Inference examplemessages = [{"role": "user", "content": "Explain gravity in one sentence."}]text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = tokenizer(text, return_tensors="pt").to(model.device) with torch.no_grad(): outputs = model.generate(**inputs, max_new_tokens=50) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))bitsandbytes) for training, adapter weights were merged with FP16 base student.import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer model_name = "sarimahsan101/Qwen2.5-0.5B-HiddenDistilled" # Load tokenizer and merged modeltokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)model = AutoModelForCausalLM.from_pretrained( model_name, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True)model.eval() # Inference examplemessages = [{"role": "user", "content": "Explain gravity in one sentence."}]text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = tokenizer(text, return_tensors="pt").to(model.device) with torch.no_grad(): outputs = model.generate(**inputs, max_new_tokens=50) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))