GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
robinzixuan
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
robinzixuan
Model Tree
Qwen/Qwen2.5-1.5B-Instruct
Input Modalities
Output Modalities
Supported Functionality
Not evaluated. No GSM8K accuracy was measured for this checkpoint, so there is no evidence here that it improves over the base model. Benchmark it before relying on it.
Loads as a plain Qwen2ForCausalLM — the run used thinking_use_mlp: false, so the checkpoint
contains no modules beyond the base architecture.
python
from transformers import AutoModelForCausalLM, AutoTokenizermodel_id = "robinzixuan/LT-Tuning-Qwen2.5-1.5B-Instruct-GSM8K"tok = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto")messages = [{"role": "user", "content": "Natalia sold clips to 48 friends in April, ""and then she sold half as many clips in May. ""How many clips did she sell altogether?"}]inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)print(tok.decode(model.generate(inputs, max_new_tokens=512)[0], skip_special_tokens=True))
Three stages of 1 epoch each, run back-to-back with the optimizer reset between stages:
| Stage | Name | Mode | Thinking insertion prob |
|---|---|---|---|
| 0 | stage0-cot | common | 0.0 |
| 1 | stage1-hidden-state | hidden_state | 0.85 |
| 2 | stage2-soft-fusion | soft_fusion | 0.95 |
Between stages the dataset is regenerated: every training example is scored with a forward pass
(thinking_strategy: confidence) to pick where the <thinking> tokens go.
| Hardware | 8× H100 80GB |
| Precision | bf16, FlashAttention-2 |
| Global batch | 32 (4 per device × 8 GPUs × 1 grad accum) |
| Steps | 702 (234 per stage) |
| LR | 5e-5, warmup ratio 0.05, weight decay 0.01 |
| Grad clipping | 1.0 |
| Wall clock | 4h08m |
| Train data | GSM8K socratic, 7473 train / 1319 val |
Final stage2-soft-fusion loss settled around 0.36 (grad_norm ~1.7, stable throughout).
The reported train_loss of 1.297 is the mean across all three stages and is not a meaningful
convergence figure — the stages optimize different objectives, so their losses are not comparable
to each other.
Qwen2.5-1.5B-Instruct base model.