Training
- Recipe:
configs/llm/deepseek_r1_distill_llama_8b_nvfp4_qad_w4a4.yaml at QATFactory commit 4d43e52869716b3445dad106d8466f245eb32a66 (main), run on 4 GPUs with --train.gradient_accumulation_steps 4 (same global batch 16 as the recipe's 8 GPUs x 2), --model.attn_implementation sdpa, checkpoints every 512 steps; the last job resumed with the resume fixes now in QATFactory (2c6a70c).
- Data:
qatfactory/openperfectblend-100k-deepseek-r1-distill-llama-8b-think at revision 62a84d9dd1dbf6aca0c9db7ce4583d553fb7bc08 (file sha256 bb7b7e5be768511c94daf14d3b8739c4d1dd6457aa200aa663a9167131f32e2a).
- Hardware and software: 1 node of 4x NVIDIA GB300, image vllm/vllm-openai:v0.30.0-aarch64 (sha256:4864d46625cbc3307623e29ac742030655e27249feba7b97ec925ce4cc4dfb56): torch 2.13.0+cu130, transformers 5.17.0, accelerate 1.15.0, plus flash-linear-attention 0.6.0 and datasets 5.0.0.
- Run: one epoch, 6,030 optimizer steps at global batch 16, lr 1e-6 cosine, forward KL to the frozen BF16 teacher, 8K context, weights and activations fake-quantized to NVFP4; trained over several 4-hour jobs, each resuming from the last checkpoint. Final held-out (128 conversations) forward KL 0.0744 and teacher-student top-1 agreement 0.898.
Evaluation
Served with vLLM 0.30.0 (data-parallel 4 on GB300, native NVFP4 W4A4, max-model-len 65536; tokenizer_class set to PreTrainedTokenizerFast, which this repo ships, since the original snapshot decodes byte tokens under transformers 5.x) and evaluated with the released harness (scripts/eval/quality/, evalscope 1.9.1, temperature 1.0, top-p 0.95, top-k 20, 32K new tokens): GPQA-Diamond, Winogrande and HellaSwag (300), MMLU-Pro and MMMLU (21 per subset), AIME25 (5 samples per problem), LiveCodeBench v6 (300) and BigCodeBench (300 of 1,140). KL: forward KL to the BF16 model over top-20 prompt logprobs, 100 samples x 8K tokens each of UltraChat, the Pile and WikiText (scripts/eval/quality/run_kld.sh). The BF16 column was measured with the same harness.
Table with columns: Benchmark, BF16, This QAD, Paper QAD (W4A4-trained)| Benchmark | BF16 | This QAD | Paper QAD (W4A4-trained) |
|---|
| GPQA-Diamond | 44.4 | 41.9 | 43.9 |
| MMLU-Pro | 53.4 | 49.3 | 48.3 |
| MMMLU | 54.4 | 54.8 | 53.4 |
| Winogrande | 69.3 | 68.3 | 68.2 |
| HellaSwag | 52.3 | 60.7 | 54.5 |
| AIME25 | 38.7 | 29.3 | 32.0 |
| LiveCodeBench v6 | 61.3 | 57.3 | 57.8 |
| BigCodeBench | 17.3 | 16.0 | 15.0 |
| Average (8) | 48.9 | 47.2 | 46.6 |
| KL UltraChat / Pile / WikiText | - | 0.0935 / 0.1239 / 0.1413 | 0.0891 / 0.1197 / 0.1356 |
| KL mean | - | 0.1196 | 0.1148 |
LiveCodeBench is graded with evalscope's local executor (no Docker on the evaluation cluster). BigCodeBench check programs are built by the harness unchanged and executed in an aarch64 rebuild of the official bigcodebench-evaluate image (all pinned requirements installed; no 4 GB memory cap; tensorflow does not import, which affects 1 of the 300 sampled tasks). Single-sample benchmarks of 198-300 questions carry 2-3 points of sampling noise; our BF16 GPQA-Diamond is 6 points below the paper's with the same protocol. The BF16 column is the same BF16 run for every checkpoint in this series.
Provenance
Files under provenance/: trainer_state.json (loss, held-out KL and top-1 history), kld-ds8b-nvfp4-w4a4-qad.json (per-dataset KL), versions.txt (training environment).