QUASAR-QAT

Qwen3.5-4B-QUASAR-NVFP4

README

License: apache-2.0

Head-to-head with every public 4-bit build of Qwen3.5-4B

One harness for every row, following RedHatAI's published protocol: 0-shot chat, thinking on, T=1.0 / top-p 0.95, 65,536-token budget, lm-evaluation-harness + lighteval, vLLM 0.26 on H100. AIME'25 is mean ± s.d. over 8 seeds, GPQA-Diamond over 3, the rest 1 seed. Bold = best 4-bit build in the row.

Table with columns: This model, RedHatAI W4A16, cyankiwi AWQ-4bit, cosmicproc NVFP4, BF16 original
	This model	RedHatAI W4A16	cyankiwi AWQ-4bit	cosmicproc NVFP4	BF16 original
Checkpoint size	4.2 GB	5.5 GB	4.0 GB	3.3 GB†	9.3 GB
Decoder projections at 4 bit	200 / 200	128 / 200	200 / 200	200 / 200	0
Vision tower · MTP drafter	kept · kept, bit-identical	kept · kept	kept · quantized	dropped · dropped	—
KL to BF16, per token ↓	.0181	.0487	.0419	.0429	0
Top-1 agreement with BF16 ↑	96.2%	93.1%	93.9%	93.7%	100%
MMLU-Pro	78.5	77.5	77.6	77.9	78.9
GSM8K-Platinum	94.7	94.0	93.9	94.2	94.0
MATH-500	83.8	83.0	84.2	84.6	84.4
AIME'25	74.2 ±3.9	65.0 ±2.5	72.5 ±4.3	65.8 ±6.1	79.2 ±5.0
GPQA-Diamond	77.9 ±1.1	71.2 ±1.8	74.6 ±3.4	73.6 ±1.1	77.3 ±1.7
IFEval, prompt / instruction	87.2 / 90.8	87.1 / 90.8	87.6 / 91.2	88.2 / 91.5	88.0 / 91.4
Hard-task recovery*	98.2%	92.9%	96.8%	94.7%	100%
gsm8k, raw 5-shot completion (no chat template)	74.8	78.2	77.4	68.1	75.2
Decode speed, 1 stream, H100	298 tok/s	239	306	230	221
Accuracy numbers published by the author	full card	partial	none	none	—

*Mean of AIME'25, GPQA-Diamond, MATH-500 and IFEval (prompt-strict) as % of the BF16 row; the y-axis of the chart. †cosmicproc ships no vision tower and no MTP drafter, and is W4A4 (needs Blackwell; measured here as W4A16 on Hopper, so its Blackwell numbers would be at most these).

Where it wins.

Fully 4-bit and fully featured. RedHatAI keeps 72 gated-delta-net projections in BF16 (+1.3 GB); cosmicproc drops the vision tower and the MTP drafter; cyankiwi quantizes the drafter. This build quantizes all 200 projections and carries vision and MTP over bit-identical.
Best 4-bit build on every hard reasoning task. AIME'25 +1.7 to +9.2 points over the other builds; GPQA-Diamond +3.3 to +6.7, above BF16; MMLU-Pro +0.6 to +1.0; GSM8K-Platinum the best of all rows, BF16 included. The only build within seed noise of BF16 on both AIME'25 and GPQA-Diamond; the others lose 7–14 points on AIME'25.
Closest to the original, token by token. Per-token KL to BF16 is 2.3–2.7× lower than every other build (.0181 vs .0419–.0487) and top-1 agreement 2.3–3.1 points higher; the other builds sit where plain rounding at our scope sits (.0423). The saturated tasks (IFEval, MATH-500) are within one point of the best build and of BF16.
Fast and small in serving: +35% single-stream decode over BF16, 25–30% faster than the INT4-GPTQ and cosmicproc builds and within 3% of the AWQ build; 3.29 GiB of weights in VRAM with the vision tower and drafter included.
Behaves like the original on plain completion prompts (raw 5-shot gsm8k 74.8 vs 75.2 BF16) — a property QAT builds distilled on chat data alone lose (61.7 for our own chat-only run); the PTQ builds keep it by construction.
Fully documented. Fidelity, benchmark, serving and training numbers on the card, bit-exact export verification; two of the three other builds publish no accuracy numbers.

For reference, RedHatAI's own card reports under its protocol (3–8 seeds): BF16 94.5 / 79.3 / 88.3 / 84.5 / 82.2 / 79.6 and its W4A16 94.5 / 77.5 / 86.1 / 83.7 / 73.3 / 73.1 (GSM8K-Plat / MMLU-Pro / IFEval / MATH-500 / AIME'25 / GPQA-D); our run of their build reproduces their MMLU-Pro exactly. No-answer rates on AIME'25 (thinking budget exhausted): BF16 17% · this model 21% · cyankiwi 19% · cosmicproc 29% · RedHatAI 33%.

Run it

bash
pip install "vllm>=0.26"

vllm serve QUASAR-QAT/Qwen3.5-4B-QUASAR-NVFP4 --max-model-len 262144 --reasoning-parser qwen3

Loads straight from the hub, no conversion (compressed-tensors NVFP4, weight-only / W4A16; weights load in under 2 s). Validated on these weights with vLLM 0.26.0 on H100 (Marlin W4A16 kernel): text and image input, a needle retrieved from a 200k-token prompt at the full 262k context, and MTP speculative decoding (--speculative-config '{"method":"mtp","num_speculative_tokens":2}': +27% single-stream decode, 80% draft-token acceptance). Blackwell should run it natively but is not yet validated by us.

Fidelity vs standard quantization at the same scope

Per-token forward KL(BF16 ‖ quantized) and top-1 agreement on 734 held-out prompts of the BF16 model's own responses (934k response tokens): the standard quantization methods at the same 200-projection NVFP4 scope (same data and schedule where trained), then the public builds on their own scope (each decoded to bf16 weights, so every row is scored the same way):

Table with columns: Size, KL ↓, KL p99 ↓, top-1 agree ↑
	Size	KL ↓	KL p99 ↓	top-1 agree ↑
QUASAR (this model)	4.2 GB	.0181	.126	96.2%
Standard QAT (STE), same scope / data / schedule	4.2 GB	.0228	.170	95.6%
GPTQ NVFP4, same scope (llmcompressor)	4.0 GB†	.0227	.167	95.7%

QUASAR cuts RTN's KL 2.3× and lands 20% below both GPTQ and standard QAT; the three public builds, decoded to bf16 and scored on the same prompts, all sit at RTN level or above (2.3–2.7× our KL). †Our GPTQ baseline export carries no MTP shard (−0.24 GB).

Serving

H100, vLLM 0.26, vllm bench serve, 1024-token prompts / 512-token outputs, 200 requests, weights-only numbers from the server log:

Table with columns: Weights in VRAM, KV-cache capacity, Output tok/s, c=1, c=32, c=128
	Weights in VRAM	KV-cache capacity	Output tok/s, c=1	c=32	c=128
BF16 original	7.99 GiB	1.88M tok	221	4,075	7,637
This model	3.29 GiB	2.02M tok	298	4,265	5,906
RedHatAI W4A16	4.49 GiB

The smallest full-featured artifact (vision + MTP included). Single-stream decode is +35% over BF16 and 25–30% over the INT4-GPTQ and cosmicproc builds; the AWQ build is 3% faster. At high batch on Hopper the Marlin NVFP4 path is compute-bound and trails BF16 / INT4 by about 20%; a W4A4 build for Blackwell is planned.

What is quantized

NVFP4 (E2M1 weights, group 16, FP8-E4M3 group scales under one fp32 tensor scale) on every large projection in every decoder layer: MLP gate/up/down in all 32 layers, q/k/v/o in the 8 full-attention layers, and in_proj_qkv / in_proj_z / out_proj in the 24 gated delta-net layers — 200 of 200. Kept in BF16: embeddings (tied), norms, the 24 depthwise conv1d kernels, the 48 small delta-net gate projections (0.11% of the decoder), the vision tower and the MTP drafter (bit-identical to the original).

Rounding the delta-net projections without training costs fidelity (RTN at this scope: KL .0423); QUASAR trains them and cuts that to .0181 while keeping the hard-task benchmarks within noise of the original. To our knowledge this is the only 4-bit build of this model at full-decoder scope that ships a fidelity and benchmark table.

The checkpoint is bit-exact: the packed NVFP4 weights decode to exactly the weights that were trained and evaluated (222,822,400 / 222,822,400 groups, 0 flipped elements), and the fused q/k/v, gate/up and in_proj_qkv/z groups share their global scale by construction, so vLLM's fused-layer loading changes nothing.

Training

Self-distillation against the frozen BF16 model: ~120M tokens seen (72k rows, 0.40 of a pass over a 296M-token corpus), global batch 32, learning rate 1e-6 (picked from a 5-point probe ladder), 2258 steps, fp32 master weights; QUASAR's loss-aware NVFP4 scale search and Fisher-weighted reconstruction inside the QAT loop. The corpus is 86.5% the BF16 model's own chat-format responses (140k rows, thinking on) and 13.5% raw web text (40k fineweb-edu documents, 40M tokens) distilled in plain completion format. The raw-text slice keeps the model's behaviour on prompts without a chat template: on 5-shot gsm8k in raw completion format it scores 74.8 vs 75.2 for BF16, where a chat-only distillation run of the same recipe drifted to 61.7 (it learned to open a <think> block the base model leaves empty in that format).

lm-evaluation-harness on vLLM against the served artifacts; zero-shot task defaults in raw completion format, GSM8K-Platinum with the chat template (0-shot CoT). avg7 = unweighted mean of the seven columns.

Table with columns: arc_c, arc_e, hellaswag, winogrande, truthfulqa, mmlu, GSM8K-Plat, avg7
	arc_c	arc_e	hellaswag	winogrande	truthfulqa	mmlu	GSM8K-Plat	avg7
BF16 original	54.4	75.5	73.1	70.4	48.8	74.3	83.9	68.6
QUASAR (this model)	54.0

Best 4-bit row, 0.6 points below BF16; differences between the 4-bit rows are mostly within noise (GSM8K-Platinum n=1209, ±1.1). This suite's IFEval column is omitted: at its 1,280-token budget it scores the raw output including <think>, which is not meaningful for a thinking model — use the head-to-head IFEval above.

Vision. The vision tower is carried over unchanged, but the distillation data was text-only, so the image path was not part of healing. Under our MMMU (validation) harness this model sits within seed noise of the BF16 original (52.4 ±2.4 vs 55.7 ±1.1 over two seeds; plain RTN at the same scope 54.4). Absolute MMMU numbers from this harness are well below Qwen's published 77.6 for all rows including BF16, so they are a relative check only; a vision-anchored revision is planned.

Known limitations

IFEval and MATH-500 are within one point of the best 4-bit build but not on top (cosmicproc leads both by 0.8–1.0, inside single-seed noise).
Weight-only quantization (W4A16): activations stay 16-bit. A W4A4 NVFP4 build for Blackwell and a GGUF build are planned; Blackwell is not yet validated by us.
The vision path was carried over, not healed: MMMU is a relative check only (see the broader-evaluation section).

QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 — the 27B sibling: 496/496 linears in NVFP4 (W4A4), GPQA-Diamond 0.909 vs 0.914 BF16.
QUASAR · Native 4-bit Qwen — the collection; Gemma 4 builds under QUASAR-QAT.
Paper: QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction.

Citation

arxiv.org/abs/2608.13966

bibtex
@article{counathe2026quasar,
  title={QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction},
  author={Counathe, Vincent and Athiwaratkun, Ben and De Sa, Christopher and Zhang, Tianyi},
  journal={arXiv preprint arXiv:2608.13966},
  year={2026}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

QUASAR-QAT

Model Tree

Base

Qwen/Qwen3.5-4B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Head-to-head with every public 4-bit build of Qwen3.5-4B

Table with columns: This model, RedHatAI W4A16, cyankiwi AWQ-4bit, cosmicproc NVFP4, BF16 original
	This model	RedHatAI W4A16	cyankiwi AWQ-4bit	cosmicproc NVFP4	BF16 original
Checkpoint size	4.2 GB	5.5 GB	4.0 GB	3.3 GB†	9.3 GB
Decoder projections at 4 bit	200 / 200	128 / 200	200 / 200	200 / 200	0
Vision tower · MTP drafter	kept · kept, bit-identical	kept · kept	kept · quantized	dropped · dropped	—
KL to BF16, per token ↓	.0181	.0487	.0419	.0429	0
Top-1 agreement with BF16 ↑	96.2%	93.1%	93.9%	93.7%	100%
MMLU-Pro	78.5	77.5	77.6	77.9	78.9
GSM8K-Platinum	94.7	94.0	93.9	94.2	94.0
MATH-500	83.8	83.0	84.2	84.6	84.4
AIME'25	74.2 ±3.9	65.0 ±2.5	72.5 ±4.3	65.8 ±6.1	79.2 ±5.0
GPQA-Diamond	77.9 ±1.1	71.2 ±1.8	74.6 ±3.4	73.6 ±1.1	77.3 ±1.7
IFEval, prompt / instruction	87.2 / 90.8	87.1 / 90.8	87.6 / 91.2	88.2 / 91.5	88.0 / 91.4
Hard-task recovery*	98.2%	92.9%	96.8%	94.7%	100%
gsm8k, raw 5-shot completion (no chat template)	74.8	78.2	77.4	68.1	75.2
Decode speed, 1 stream, H100	298 tok/s	239	306	230	221
Accuracy numbers published by the author	full card	partial	none	none	—

Where it wins.

Fully 4-bit and fully featured. RedHatAI keeps 72 gated-delta-net projections in BF16 (+1.3 GB); cosmicproc drops the vision tower and the MTP drafter; cyankiwi quantizes the drafter. This build quantizes all 200 projections and carries vision and MTP over bit-identical.
Best 4-bit build on every hard reasoning task. AIME'25 +1.7 to +9.2 points over the other builds; GPQA-Diamond +3.3 to +6.7, above BF16; MMLU-Pro +0.6 to +1.0; GSM8K-Platinum the best of all rows, BF16 included. The only build within seed noise of BF16 on both AIME'25 and GPQA-Diamond; the others lose 7–14 points on AIME'25.
Closest to the original, token by token. Per-token KL to BF16 is 2.3–2.7× lower than every other build (.0181 vs .0419–.0487) and top-1 agreement 2.3–3.1 points higher; the other builds sit where plain rounding at our scope sits (.0423). The saturated tasks (IFEval, MATH-500) are within one point of the best build and of BF16.
Fast and small in serving: +35% single-stream decode over BF16, 25–30% faster than the INT4-GPTQ and cosmicproc builds and within 3% of the AWQ build; 3.29 GiB of weights in VRAM with the vision tower and drafter included.
Behaves like the original on plain completion prompts (raw 5-shot gsm8k 74.8 vs 75.2 BF16) — a property QAT builds distilled on chat data alone lose (61.7 for our own chat-only run); the PTQ builds keep it by construction.
Fully documented. Fidelity, benchmark, serving and training numbers on the card, bit-exact export verification; two of the three other builds publish no accuracy numbers.

Run it

bash
pip install "vllm>=0.26"

vllm serve QUASAR-QAT/Qwen3.5-4B-QUASAR-NVFP4 --max-model-len 262144 --reasoning-parser qwen3

Fidelity vs standard quantization at the same scope

Table with columns: Size, KL ↓, KL p99 ↓, top-1 agree ↑
	Size	KL ↓	KL p99 ↓	top-1 agree ↑
QUASAR (this model)	4.2 GB	.0181	.126	96.2%
Standard QAT (STE), same scope / data / schedule	4.2 GB	.0228	.170	95.6%
GPTQ NVFP4, same scope (llmcompressor)	4.0 GB†	.0227	.167	95.7%

Serving

H100, vLLM 0.26, vllm bench serve, 1024-token prompts / 512-token outputs, 200 requests, weights-only numbers from the server log:

Table with columns: Weights in VRAM, KV-cache capacity, Output tok/s, c=1, c=32, c=128
	Weights in VRAM	KV-cache capacity	Output tok/s, c=1	c=32	c=128
BF16 original	7.99 GiB	1.88M tok	221	4,075	7,637
This model	3.29 GiB	2.02M tok	298	4,265	5,906
RedHatAI W4A16	4.49 GiB

What is quantized

Training

Table with columns: arc_c, arc_e, hellaswag, winogrande, truthfulqa, mmlu, GSM8K-Plat, avg7
	arc_c	arc_e	hellaswag	winogrande	truthfulqa	mmlu	GSM8K-Plat	avg7
BF16 original	54.4	75.5	73.1	70.4	48.8	74.3	83.9	68.6
QUASAR (this model)	54.0

Known limitations

IFEval and MATH-500 are within one point of the best 4-bit build but not on top (cosmicproc leads both by 0.8–1.0, inside single-seed noise).
Weight-only quantization (W4A16): activations stay 16-bit. A W4A4 NVFP4 build for Blackwell and a GGUF build are planned; Blackwell is not yet validated by us.
The vision path was carried over, not healed: MMMU is a relative check only (see the broader-evaluation section).

QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 — the 27B sibling: 496/496 linears in NVFP4 (W4A4), GPQA-Diamond 0.909 vs 0.914 BF16.
QUASAR · Native 4-bit Qwen — the collection; Gemma 4 builds under QUASAR-QAT.
Paper: QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction.

Citation

arxiv.org/abs/2608.13966

bibtex
@article{counathe2026quasar,
  title={QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction},
  author={Counathe, Vincent and Athiwaratkun, Ben and De Sa, Christopher and Zhang, Tianyi},
  journal={arXiv preprint arXiv:2608.13966},
  year={2026}
}

Qwen3.5-4B-QUASAR-NVFP4

README

Head-to-head with every public 4-bit build of Qwen3.5-4B

Run it

Fidelity vs standard quantization at the same scope

Serving

What is quantized

Training

Known limitations

Related

Citation

Explore FriendliAI today

README

Head-to-head with every public 4-bit build of Qwen3.5-4B

Run it

Fidelity vs standard quantization at the same scope

Serving

What is quantized

Training

Known limitations

Related

Citation