Evaluation
Thinking mode, pass@1 averaged over 1–32 seeds. Same sampling, token caps and seeds for every model; the ParoQuant column ran on its own serving stack (vLLM 0.19.1 with paroquant 0.1.16); MMLU-Pro answers are extracted with the
official MMLU-Pro lenient rules, because this model does not follow the standard answer format.
Speed on one H100 80GB, 512 generated tokens per request (tok/s = generated tokens / wall time, mean of 2 runs): 297 tok/s at batch 1 and 4,245 at batch 16; ParoQuant 149 and 2,275 (both on vLLM 0.19.1).
Usage
Runs on NVIDIA H100, A100, RTX Pro 6000 and B200 GPUs.
vLLM:
vllm serve qda-lab/QDA-DeepSeek-R1-Distill-Llama-8B-W4A16
SGLang:
python -m sglang.launch_server --model-path qda-lab/QDA-DeepSeek-R1-Distill-Llama-8B-W4A16
How it was made
Weights are 4-bit integers in groups of 128, quantized with GPTQ on a Hadamard-rotated copy of the model; the rotation is folded
into the weights, so there is no runtime transform. The group scales were then fine-tuned by distillation. The recipe was chosen by QDA's automated search.
Reproduce
hf download qda-lab/qda-recipe-inputs --repo-type dataset --include "QDA-DeepSeek-R1-Distill-Llama-8B-W4A16/*" --local-dir inputsqda build inputs/QDA-DeepSeek-R1-Distill-Llama-8B-W4A16/r1-llama-8b-w4a16.yaml
Needs the QDA library (the qda command). Recipe: r1-llama-8b-w4a16.yaml; inputs
(calibration sample and distillation data): QDA-DeepSeek-R1-Distill-Llama-8B-W4A16/ in qda-lab/qda-recipe-inputs.
Takes about 2.8 h on 1× H100 (measured). The rebuild matches this checkpoint's KL to BF16 within 5 %; weights are not bit-identical.
License
MIT, the same as the base model.
Citation
TBD.