Models
Table with columns: Role, Model| Role | Model |
|---|
| Student | Qwen3-4B-Instruct-2507 (after stages 1–3 above) |
| Teacher | Qwen/Qwen3-30B-A3B-Instruct-2507 (MoE, ~3B active) |
enable_thinking=False throughout.
Training data
Note on data overlap. The parent openthoughts3_math_50k[30k:40k] window contains 418 internal duplicate prompts across its two halves, so ~8.4% of the 5k prompts here also appeared in the prior stage-3 slice (30k–35k). This is an upstream data artifact of the OpenThoughts-3 math split, not from the slicing; the file itself is still exactly parent[5000:10000].
Training setup
Framework: KDFlow — FSDP2 + SGLang rollout, Ray-orchestrated GPU co-location with sleep/wakeup.
Hardware: 1× node, 4× H100 (80 GB), ~34 h wall-clock (~136 GPU-hours).
Key hyperparameters (identical to stages 2 and 3)
Table with columns: Group, Value| Group | Value |
|---|
| Backend | fsdp2, bf16, gradient ckpt on |
| Epochs | 1 |
| Train batch | 4 (micro 1) |
| Learning rate | 2e-6, cosine, warmup 5% |
| KD ratio | 1.0 |
| KD loss | reverse KL (rkl) |
| KD algorithm | vanilla_kd |
| Temperature (KD) | 1.0 |
Intended use
Research on distillation dynamics and the effect of successive continual on-policy rounds on the same student. Domain: math (OpenThoughts-3 math split).
Limitations
- Trained end-to-end on math prompts only; not tuned for chat, safety, or non-math domains.
enable_thinking=False — this student does not emit <think> traces.- Chain-of-continual training may drift capabilities relative to earlier checkpoints in the chain; see the wandb curves for loss dynamics.