GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
QUASAR-QAT
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
QUASAR-QAT
Model Tree
Qwen/Qwen3.5-4B
Input Modalities
Output Modalities
Supported Functionality
One harness for every row, following RedHatAI's published protocol (0-shot chat, thinking on, 65,536-token budget, vLLM 0.26 on H100). AIME'25 is the mean of 8 seeds, GPQA-Diamond of 3. Bold = best among the compared 4-bit builds.
| QUASAR (this model) | RedHatAI W4A16 | cyankiwi AWQ-4bit | cosmicproc NVFP4 | BF16 original | |
|---|---|---|---|---|---|
| Size | 4.2 GB | 5.5 | 4.0 | 3.3 | 9.3 |
| Decoder projections @ 4-bit | 200 / 200 | 128 / 200 | 200 / 200 | 200 / 200 | — |
| Vision / MTP | ✓ / ✓ | ✓ / ✓ | ✓ / quantized | ✗ / ✗ | ✓ / ✓ |
| KL to BF16 ↓ | .0181 | .0487 | .0419 | .0429 | — |
| MMLU-Pro ↑ | 78.5 | 77.5 | 77.6 | 77.9 | 78.9 |
| AIME'25 ↑ | 74.2 | 65.0 | 72.5 | 65.8 | 79.2 |
| GPQA-Diamond ↑ | 77.9 | 71.2 | 74.6 | 73.6 | 77.3 |
| GSM8K-Platinum ↑ | 94.7 | 94.0 | 93.9 | 94.2 | 94.0 |
| H100 decode, c=1 ↑ | 298 tok/s | 239 | 306 | 230 | 221 |
Full benchmark table, seeds, confidence information, fidelity methodology and serving measurements: EVAL.md.
bash
pip install "vllm>=0.26" vllm serve QUASAR-QAT/Qwen3.5-4B-QUASAR-NVFP4 --max-model-len 262144 --reasoning-parser qwen3Loads straight from the Hub, no conversion (compressed-tensors NVFP4, weight-only / W4A16). Validated with vLLM 0.26 on H100 for text, image input, 262K context (needle retrieved from a 200k-token prompt) and MTP speculative decoding (--speculative-config '{"method":"mtp","num_speculative_tokens":2}': +27% single-stream decode, 80% draft acceptance). For Blackwell, use the W4A4 build; for llama.cpp / Ollama, the Q4_0 GGUF.
Per-token forward KL(BF16 ‖ quantized) and top-1 agreement on 734 held-out prompts of the BF16 model's own responses (934k response tokens), all methods at the same 200-projection NVFP4 scope (same data and schedule where trained):
| KL ↓ | KL p99 ↓ | top-1 agree ↑ | |
|---|---|---|---|
| QUASAR (this model) | .0181 | .126 | 96.2% |
| Standard QAT (STE) | .0228 | .170 | 95.6% |
| GPTQ NVFP4 (llmcompressor) | .0227 | .167 | 95.7% |
| RTN NVFP4 (no training) | .0423 | .326 | 93.8% |
At identical NVFP4 scope, QUASAR cuts RTN KL by 2.3× and improves ~20% over both GPTQ and standard QAT.
NVFP4 (E2M1 weights, group 16, FP8-E4M3 group scales under one fp32 tensor scale) on every large projection in every decoder layer: MLP gate/up/down in all 32 layers, q/k/v/o in the 8 full-attention layers, and in_proj_qkv / in_proj_z / out_proj in the 24 gated delta-net layers — 200 of 200. Kept in BF16: embeddings (tied), norms, the 24 depthwise conv1d kernels, the 48 small delta-net gate projections (0.11% of the decoder), the vision tower and the MTP drafter (bit-identical to the original). Full-decoder NVFP4, with vision and MTP retained.
The packed NVFP4 weights decode to exactly the weights that were trained and evaluated (222,822,400 / 222,822,400 groups, 0 flipped elements — export_ct_report.json); fused q/k/v, gate/up and in_proj_qkv/z groups share their global scale by construction, so vLLM's fused-layer loading changes nothing.
Training. QUASAR QAT with self-distillation from the frozen BF16 checkpoint over ~120M tokens; QUASAR performs loss-aware NVFP4 reconstruction inside the training loop. The exported packed weights are bit-exact to those trained and evaluated. Details in EVAL.md and the paper.
QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 — the 27B sibling: 496/496 linears in NVFP4 (W4A4), GPQA-Diamond 0.909 vs 0.914 BF16.QUASAR-QAT/Qwen3.5-4B-QUASAR-NVFP4-W4A4 — the same weights with calibrated activation scales for native W4A4 on Blackwell.QUASAR-QAT/Qwen3.5-4B-QUASAR-Q4_0-GGUF — native Q4_0 QAT for llama.cpp / Ollama / LM Studio.bibtex
@article{counathe2026quasar, title={QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction}, author={Counathe, Vincent and Athiwaratkun, Ben and De Sa, Christopher and Zhang, Tianyi}, journal={arXiv preprint arXiv:2608.13966}, year={2026}}pip install "vllm>=0.26" vllm serve QUASAR-QAT/Qwen3.5-4B-QUASAR-NVFP4 --max-model-len 262144 --reasoning-parser qwen3