Quick facts
Table | |
|---|
| Base model | Qwen/Qwen3.6-35B-A3B |
| Architecture | Qwen3_5MoeForConditionalGeneration — 40 layers, 10 × (3 × Gated DeltaNet + 1 × Gated Attention), each followed by an MoE block with 256 experts (8 routed + 1 shared), hidden 2048, 262K context, vision encoder, 1 MTP layer |
| Original size | 67 GB (BF16) |
| Quantized size | 25.4 GB (5 shards) |
| Quant format | NVFP4 via nvidia-modelopt 0.43.0, group_size=16, quant_algo: W4A16_NVFP4 |
| Quantized modules | 30,840 Linear layers: per layer 256 experts × {gate,up,down}_proj + shared_expert.{gate,up,down}_proj |
| Kept BF16 | self_attn*, linear_attn* (Gated DeltaNet), mlp.gate (router), mlp.shared_expert_gate, embed_tokens, lm_head, all norms, vision tower (333 tensors), MTP head (19 tensors) |
| Calibration | 128 × 1024 tokens of abisee/cnn_dailymail; only used for the shipped input_scale (see below) |
| Conversion date | 2026-11-15 |
Quality: KLD against BF16
Top-50 KLD of the next-token distribution against the BF16 model, both served by vLLM 0.31.0 on one GB10. 72 held-out sequences, 276,638 token positions: news, Wikipedia EN/DE, code, chat, and two roleplay sets (private, not published).
Table with columns: Variant, mean, median, p90, p99| Variant | mean | median | p90 | p99 |
|---|
| this repo (W4A16) | 0.0758 | 0.0065 | 0.096 | 1.17 |
| nvidia/Qwen3.6-35B-A3B-NVFP4 | 0.0998 | 0.0124 | 0.145 | 1.53 |
same weights as W4A4 (quant_algo: NVFP4) | 0.1103 | 0.0122 | 0.171 | 1.83 |
Measured on the files in this repo. Repeated W4A16 runs of the same weights scatter by about ±0.0005 (vLLM is not bit-deterministic across runs), so differences below ~0.001 are noise.
Per bucket (mean KLD):
Table with columns: Bucket, this repo, NVIDIA, W4A4| Bucket | this repo | NVIDIA | W4A4 |
|---|
| news | 0.050 | 0.070 | 0.075 |
| wiki_en | 0.051 | 0.066 | 0.088 |
| wiki_de | 0.025 | 0.035 | 0.043 |
| code | 0.029 | 0.036 | 0.048 |
| chat | 0.044 | 0.058 | 0.064 |
| roleplay A | 0.050 | 0.070 | 0.082 |
| roleplay B | 0.229 | 0.296 | 0.297 |
Where the difference to NVIDIA's release comes from: the expert weights are bit-identical to NVIDIA's, and both releases serve them as W4A16. The difference is everything around them — NVIDIA stores the attention and Gated DeltaNet projections in FP8, lm_head in NVFP4, and declares an FP8 KV cache; this repo keeps all of them in BF16 (that is the extra ~2 GB). Both were measured as shipped, with vLLM defaults (no --kv-cache-dtype override). Which of the three parts costs how much was not measured separately.
Why W4A16 and not W4A4: a plain NVFP4 config makes vLLM 0.31 on GB10 quantize the activations to FP4 as well (FlashInferCutlassNvFp4LinearKernel). That costs ~45 % more KLD for no decode gain. The activation scale is not the problem: calibrated input_scale, input_scale = 1.0 and NVIDIA's scales all measured 0.109–0.110 — it is the FP4 rounding of the activations itself.
Speed (single DGX Spark, GB10, vLLM 0.31.0)
Table with columns: W4A16 (this repo), W4A4 (same weights) | W4A16 (this repo) | W4A4 (same weights) |
|---|
| Prefill, ~4k-token prompt | 5,969 tok/s | 6,562 tok/s |
| Decode, 1 stream | 38.7 tok/s | 37.9 tok/s |
| Decode, 4 streams (total) | 132.8 tok/s | 126.6 tok/s |
| Decode, 1 stream, MTP k=1 | 58.0 tok/s | – |
| Decode, 1 stream, MTP k=2 | 57.2 tok/s | – |
| Decode, 4 streams (total), MTP k=1 | 171.5 tok/s | – |
GEMM path: MarlinNvFp4LinearKernel. MTP draft acceptance: 84 % at k=1 (mean acceptance length 1.84, +50 % single-stream decode); k=2 accepts 74 % / 54 % per position (length 2.27) and is not faster on GB10, so k=1 is recommended. Prefill is unaffected by MTP (5,947 tok/s).
Verification
On a single DGX Spark (GB10), vLLM 0.31.0:
- loads without missing-parameter warnings; MTP head detected and used (
quantization=modelopt_fp4, MarlinNvFp4LinearKernel)
- model memory 23.8 GiB (incl. MTP); at
--gpu-memory-utilization 0.5 (64 GB of the 128 GB UMA pool) 21.0 GiB are left for KV cache = 620,625 tokens. Only 10 of 40 layers (plus MTP) carry a KV cache.
- coherent English and German output (tested with thinking off), correct arithmetic, working Python code, in-character roleplay replies
- KLD re-measured on the released files (table above)
Vision was not re-tested after quantization; the vision tower is untouched BF16.
Usage
vLLM (serve)
vllm serve Kaleto/Qwen3.6-35B-A3B-NVFP4 \ --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --attention-backend flashinfer \ --max-model-len 262144 \ --max-num-seqs 4 \ --gpu-memory-utilization 0.5 \ --enable-prefix-caching
- No
--quantization flag needed; vLLM reads the modelopt config.
--gpu-memory-utilization is a fraction of total memory; 0.5 on a 128 GB Spark leaves room for other models. Scale it to your GPU.
- Drop
--speculative-config to serve without MTP; the MTP tensors are then ignored.
- Text only: add
--language-model-only to skip the vision encoder and free memory for KV cache.
W4A4 instead: the calibrated input_scale tensors are included. Set quant_algo to NVFP4 in config.json and hf_quant_config.json to get the W4A4 path (faster prefill, see the tables above for the quality cost).
Sampling
Qwen's recommendations (all with top_k=20, min_p=0, repetition_penalty=1.0):
Table with columns: Mode, temperature, top_p, presence_penalty| Mode | temperature | top_p | presence_penalty |
|---|
| Thinking, general | 1.0 | 0.95 | 1.5 |
| Thinking, precise coding | 0.6 | 0.95 | 0.0 |
| Non-thinking | 0.7 | 0.8 | 1.5 |
Toggle thinking per request with chat_template_kwargs: {"enable_thinking": false}. min_p and logit_bias have no effect while speculative decoding is active (vLLM limitation).
How it was made
distrib-nvfp4, sequential mode: one decoder layer at a time is loaded in BF16, quantized with NVFP4_DEFAULT_CFG, calibrated on the hidden states of the previous layer and exported. Peak memory is about one layer plus the hidden states, so a single Spark handles any model size; with several nodes the calibration samples are split across them. During calibration every token is routed to all 256 experts, so every expert gets real statistics. The vision tower and MTP head are copied verbatim from the source checkpoint.
Checks done on this artifact: all 30,840 quantized modules carry a calibrated input_scale; a resumed and a distributed (2-shard) run of the same pipeline produced bit-identical files.
Files
model-0000{1..5}-of-00005.safetensors — NVFP4 expert weights + scales, BF16 for everything else, vision tower and MTP head
model.safetensors.index.json — 124,205 tensors
config.json — upstream config + quantization_config (W4A16_NVFP4, 123 ignore entries)
hf_quant_config.json, generation_config.json
- tokenizer, chat template and processor files — unchanged from upstream
Acknowledgments
- Qwen team for Qwen3.6-35B-A3B
- NVIDIA for DGX Spark / GB10, NVFP4 and modelopt, and for publishing their own NVFP4 release as a reference
- vLLM project for modelopt NVFP4, Marlin FP4 and Qwen3.6 MTP support
License
Apache 2.0, inherited from Qwen/Qwen3.6-35B-A3B.