Evaluation
Thinking mode, pass@1 averaged over 1–32 seeds, same settings for all models; MMLU-Pro answers are extracted with the
official MMLU-Pro lenient rules, because this model does not follow the standard answer format.
Usage
Runs on vLLM on NVIDIA Hopper GPUs only (H100 / H200, sm90): the W4A8 kernel in vLLM requires compute capability 9.0.
vLLM:
vllm serve qda-lab/QDA-DeepSeek-R1-Distill-Llama-8B-W4A8
How it was made
Weights are 4-bit integers in groups of 128 and activations are 8-bit floating point (dynamic), quantized with GPTQ on a Hadamard-rotated copy of the model; the rotation is folded
into the weights, so there is no runtime transform. The recipe was chosen by QDA's automated search.
Reproduce
hf download qda-lab/qda-recipe-inputs --repo-type dataset --include "QDA-DeepSeek-R1-Distill-Llama-8B-W4A8/*" --local-dir inputscd inputs/QDA-DeepSeek-R1-Distill-Llama-8B-W4A8qda build r1-llama-8b-w4a8.yaml # part 1qda build r1-llama-8b-w4a8-long2.yaml # part 2 (can run beside part 3, on another card)qda build r1-llama-8b-w4a8-long.yaml # part 3qda build r1-llama-8b-w4a8-soup.yaml # soup of parts 1-3 -> ./QDA-DeepSeek-R1-Distill-Llama-8B-W4A8-rebuild
Needs the QDA library (the qda command). Recipes and inputs (calibration sample and distillation data):
QDA-DeepSeek-R1-Distill-Llama-8B-W4A8/ in qda-lab/qda-recipe-inputs.
Takes about 13 h on H100s (measured: part 1 8.7 h on one card, then parts 2 and 3 side by side on two cards, 4.3 h; the soup takes seconds on CPU). The rebuild matches this checkpoint's KL to BF16 within 5 %; weights are not bit-identical.
License
MIT, the same as the base model.
Citation
TBD.