peterhdd

Qwen3.6-35B-A3B-NVFP4A16

README

License: apache-2.0

Results

All numbers measured on the same machine with identical settings: 1× RTX PRO 6000 Blackwell (96 GB), vLLM 0.32.0, tensor parallel 1.

Table with columns: Build, Size, MMLU-Pro, Recovery, GSM8K, Recovery, 1-user tok/s, 16-user tok/s
Build	Size	MMLU-Pro	Recovery	GSM8K	Recovery	1-user tok/s	16-user tok/s
Qwen/Qwen3.6-35B-A3B (BF16)	67 GiB	0.837	100.0%	0.921	100.0%	147.6	793.7
This model (NVFP4A16)	24 GiB	0.831	99.2%	0.923	100.2%	173.2	1,394.8
nvidia/Qwen3.6-35B-A3B-NVFP4	22 GiB	0.831	99.2%	0.926	100.6%	207.5	1,507.5
kyaky/Qwen3.6-35B-A3B-NVFP4	21 GiB	0.835	99.7%	0.929	100.8%	188.9	1,484.0
RedHatAI/Qwen3.6-35B-A3B-NVFP4	24 GiB	0.829	99.1%	0.927	100.7%	158.8	1,330.5

Margins of error: about ±0.010 on MMLU-Pro and ±0.007 on GSM8K.

How to read this:

Quality is a tie. Every build, including this one, lands within the margin of error of BF16 and of each other. No NVFP4 build here is measurably more accurate than another.
Size and speed differ. nvidia's build is the fastest and kyaky's the smallest; this build is faster than RedHatAI's but larger and slower than nvidia's and kyaky's. If speed or size matters most to you, those builds are the better pick on this GPU.
Answers with no final response (the model thought until the limit without answering) were rare for every build: 0–2 of 1,400 on MMLU-Pro, and 7–10 of 1,319 on GSM8K where measured, the same rate as BF16.
Single-user speed was measured on only 8 requests, so differences of 5–10% in that column are within run-to-run variation.

Recovery % is a build's score divided by the BF16 score.

Quantization recipe

Routed and shared MoE experts, full-attention projections: NVFP4 weights (FP4 E2M1 values, one FP8 E4M3 scale per 16 weights, plus a per-tensor scale), 16-bit activations (W4A16)
Kept in BF16: lm_head, embeddings, MoE router gates, shared-expert gate, linear-attention (GatedDeltaNet) layers, vision tower
MTP layers: copied unchanged from the original checkpoint
Calibration: none needed. Weight-only NVFP4 scales are computed directly from the weights (llm-compressor's data-free pipeline)

Why weight-only on this GPU

On sm120 (RTX PRO 6000 / RTX 50-series Blackwell), vLLM serves NVFP4 MoE experts through the Marlin W4A16 kernel path, so quantizing activations to FP4 too (W4A4) brings no speed gain there, only extra quantization error. Weight-only keeps quality closer to the original at the same speed.

Usage

bash

vllm serve peterhdd/Qwen3.6-35B-A3B-NVFP4A16 \  --reasoning-parser qwen3 \  --max-num-seqs 256

Add --language-model-only to skip loading the vision tower when you only need text.

The model thinks by default (<think>…</think>); --reasoning-parser qwen3 separates the thinking from the final answer. vLLM selects the Marlin NVFP4 kernels automatically on Blackwell.

Hardware: tested only on NVIDIA RTX PRO 6000 Blackwell (sm120). Weight-only NVFP4 runs through vLLM's Marlin kernels, which may also support some older GPUs, but that hasn't been tested.

Evaluation details

Harness: lm-evaluation-harness via vLLM's OpenAI-compatible chat endpoint, thinking on, chat template applied, 0-shot
MMLU-Pro: task mmlu_pro_llama, 100 questions per subject (1,400 total), strict-match
GSM8K: task gsm8k_cot_llama, full test set (1,319), flexible-extract
Sampling: temperature 1.0, top_p 0.95, top_k 20, min_p 0, presence_penalty 1.5, max 32,000 new tokens, seed 42
Speed: vllm bench serve, random prompts of 1,024 input / 512 output tokens, concurrency 1 and 16
Server: --max-model-len 65536 --max-num-seqs 256 --gpu-memory-utilization 0.90

Limitations

This build keeps the linear-attention layers in BF16. kyaky's build stores attention and shared-expert layers in FP8 instead, which is likely part of why it is smaller; that variant was not tried here.
Evaluated on MMLU-Pro (subset) and GSM8K only, with a single seed. Coding, tool calling and agentic tasks, where 4-bit quantization usually loses the most, were not tested.
Vision input and very long contexts were not tested.
Benchmarked on one GPU type only.

Reproduce

Built with llm-compressor (main, 0.14.1.dev), compressed-tensors ≥ 0.19.1a2, transformers ≥ 5.

Scripts and raw results: https://github.com/PeterHdd/Qwen3.6-35B-A3B-NVFP4A16

References

Base model: Qwen/Qwen3.6-35B-A3B
Quantization recipe adapted from RedHatAI/Qwen3.6-35B-A3B-NVFP4, switched to weight-only (W4A16)
Weight-only on sm120 idea and comparison format from kyaky/Qwen3.6-35B-A3B-NVFP4
Other builds compared: nvidia/Qwen3.6-35B-A3B-NVFP4
Tools: llm-compressor, compressed-tensors, vLLM, lm-evaluation-harness
Benchmarks: MMLU-Pro, GSM8K

License

Apache 2.0, same as the base model. All credit for the model itself goes to the Qwen team.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

peterhdd

Model Tree

Base

Qwen/Qwen3.6-35B-A3B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality