Evaluation
Thinking mode, pass@1 averaged over 1–32 seeds. Same sampling, token caps and seeds for every model; the ParoQuant column ran on its own serving stack (vLLM 0.19.1 with a config override it needs to load, and the chat template shipped in its repo, which differs from Google's current template by one newline in the system turn).
Speed on one H100 80GB, 512 generated tokens per request (tok/s = generated tokens / wall time, mean of 2 runs): 95 tok/s at batch 1 and 1,101 to 1,251 at batch 16 (the two runs); ParoQuant 66 and 891 (both on vLLM 0.19.1).
Usage
Runs on NVIDIA H100, A100, RTX Pro 6000 and B200 GPUs.
vLLM:
VLLM_DISABLED_KERNELS=MacheteLinearKernel vllm serve qda-lab/QDA-Gemma-4-31B-W4A16
On H100 and H200, without a CUDA 13 toolkit on the machine, the kernel's compile at start fails (
failed to open libnvrtc-builtins.so.13.0
). Put the venv's NVIDIA libraries on the loader path first:
lib="$(python -c 'import glob, sysconfig as s; p = s.get_paths(); print(":".join(sorted({d for k in ("purelib", "platlib") for d in glob.glob(p[k] + "/nvidia/*/lib")})))')" && export LD_LIBRARY_PATH="$lib:$LD_LIBRARY_PATH"
SGLang:
python -m sglang.launch_server --model-path qda-lab/QDA-Gemma-4-31B-W4A16
How it was made
Weights are 4-bit integers in groups of 32, quantized with GPTQ; the group scales were then fine-tuned by distillation.
The vision tower stays in BF16. The recipe was chosen by QDA's automated search.
Reproduce
Install QDA 0.1.0 (Linux, an NVIDIA GPU and uv):
git clone https://github.com/humanfia/qda.git && cd qdauv venv -p 3.12 --managed-python && . .venv/bin/activateuv pip install -c constraints.txt "./packages/qda-quant[hf,gguf]" "./packages/qda-kl[vllm]" ./packages/qda-agent
Then:
hf download qda-lab/qda-recipe-inputs --repo-type dataset --include "QDA-Gemma-4-31B-W4A16/*" --local-dir inputscd inputs/QDA-Gemma-4-31B-W4A16qda build gemma-4-31b-w4a16.yaml --set out=QDA-Gemma-4-31B-W4A16-rebuild --set "devices=[0,1]"
Needs QDA 0.1.0. Original model: google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475, the revision the recipe builds from. Recipe: gemma-4-31b-w4a16.yaml; inputs
(calibration sample and distillation data): QDA-Gemma-4-31B-W4A16/ in qda-lab/qda-recipe-inputs.
Built on 2× H100 (80 GB) in 63 min. Rebuilt weights are not bit-identical to this checkpoint: scored on the same prompts, a rebuild reads KL 0.01868 against the original and this checkpoint 0.01864 (+0.2 %).
License
Apache-2.0, the same as the base model (Gemma 4 license).
Citation
QDA: Closed-Loop LLM Quantization Design with Agents
@article{li2026qda, title = {QDA: Closed-Loop LLM Quantization Design with Agents}, author = {Li, Xingyang and Zhao, Tianlang and Han, Song and Zhu, Ligeng}, journal = {arXiv preprint arXiv:2611.23388}, year = {2026}}