Models
Table with columns: Role, Model| Role | Model |
|---|
| Student | Qwen3-4B-Instruct-2507 (after stages 1–2 above) |
| Teacher | Qwen/Qwen3-30B-A3B-Instruct-2507 (MoE, ~3B active) |
enable_thinking=False throughout (both stages).
Training data
Training setup
Framework: KDFlow — FSDP2 + SGLang rollout, Ray-orchestrated GPU co-location with sleep/wakeup.
Hardware: 1× node, 4× H100 (80 GB), ~34 h wall-clock (~136 GPU-hours).
Key hyperparameters (on-policy stage; same as stage 2)
Table with columns: Group, Value| Group | Value |
|---|
| Backend | fsdp2, bf16, gradient ckpt on |
| Epochs | 1 |
| Train batch | 4 (micro 1) |
| Learning rate | 2e-6, cosine, warmup 5% |
| KD ratio | 1.0 |
| KD loss | reverse KL (rkl) |
| KD algorithm | vanilla_kd |
| Temperature (KD) | 1.0 |
Off-policy stage 1 hyperparameters (for context): batch 128, LR 2e-5, kd_ratio 0.5, kd_loss KL, max_len 16384, packing_samples on, ring attention.
Intended use
Research on distillation dynamics. Domain skew is math (OpenThoughts-3 math split).
Limitations
- Trained end-to-end on math prompts only; not tuned for chat, safety, or non-math domains.
enable_thinking=False — this student does not emit <think> traces.- Continual training may drift capabilities relative to the base checkpoint; see the wandb curves for loss dynamics.
Weights & Biases
legendary-music-16 (stage 2, 10k) and lemon-sound-17 / 2gvd60dw (this run, stage 3) under project KDFlow, entity zhao666.