Evaluation
Thinking mode, pass@1 averaged over 1–32 seeds. Same sampling, token caps and seeds for every model; the ParoQuant column ran on its own serving stack (vLLM 0.19.1 with our MoE adapter and a group-size patch, one GPU).
Speed on one H100 80GB, 512 generated tokens per request (tok/s = generated tokens / wall time, mean of 2 runs): 198 tok/s at batch 1 and 2,330 at batch 16 on vLLM 0.33.0; ParoQuant 133 and 1,652 on vLLM 0.19.1 with our MoE adapter.
Usage
Runs on NVIDIA H100, A100, RTX Pro 6000 and B200 GPUs.
vLLM:
VLLM_DISABLED_KERNELS=MacheteLinearKernel vllm serve qda-lab/QDA-Gemma-4-26B-A4B-W4A16
On H100 and H200, without a CUDA 13 toolkit on the machine, the kernel's compile at start fails (
failed to open libnvrtc-builtins.so.13.0
). Put the venv's NVIDIA libraries on the loader path first:
lib="$(python -c 'import glob, sysconfig as s; p = s.get_paths(); print(":".join(sorted({d for k in ("purelib", "platlib") for d in glob.glob(p[k] + "/nvidia/*/lib")})))')" && export LD_LIBRARY_PATH="$lib:$LD_LIBRARY_PATH"
How it was made
The routed experts are 4-bit integers in groups of 64, quantized with GPTQ; all other linear layers are 8-bit.
MoE routers and the vision tower stay in BF16. The recipe was chosen by QDA's automated search.
Reproduce
Install QDA 0.1.0 (Linux, an NVIDIA GPU and uv):
git clone https://github.com/humanfia/qda.git && cd qdauv venv -p 3.12 --managed-python && . .venv/bin/activateuv pip install -c constraints.txt "./packages/qda-quant[hf,gguf]" "./packages/qda-kl[vllm]" ./packages/qda-agent
Then:
hf download qda-lab/qda-recipe-inputs --repo-type dataset --include "QDA-Gemma-4-26B-A4B-W4A16/*" --local-dir inputscd inputs/QDA-Gemma-4-26B-A4B-W4A16qda build gemma-4-26b-a4b-w4a16.yaml --set out=QDA-Gemma-4-26B-A4B-W4A16-rebuild
Needs QDA 0.1.0. Original model: google/gemma-4-26B-A4B-it@4d7ae4984b7db7de8f8457170b3f1a419ee76d52, the revision the recipe builds from. Recipe: gemma-4-26b-a4b-w4a16.yaml; inputs
(calibration sample and distillation data): QDA-Gemma-4-26B-A4B-W4A16/ in qda-lab/qda-recipe-inputs.
Built on 1× H100 (80 GB). Rebuilt weights are not bit-identical to this checkpoint.
License
Apache-2.0, the same as the base model (Gemma 4 license).
Citation
QDA: Closed-Loop LLM Quantization Design with Agents
@article{li2026qda, title = {QDA: Closed-Loop LLM Quantization Design with Agents}, author = {Li, Xingyang and Zhao, Tianlang and Han, Song and Zhu, Ligeng}, journal = {arXiv preprint arXiv:2611.23388}, year = {2026}}