Quick facts
Table | |
|---|
| Base model | Qwen/Qwen3.5-122B-A10B |
| Architecture | Qwen3_5MoeForConditionalGeneration — 48 layers, 12 × (3 × Gated DeltaNet + 1 × Gated Attention), each followed by an MoE block with 256 experts (8 routed + 1 shared), hidden 3072, 262K context, vision encoder, 1 MTP layer |
| Original size | 250 GB (BF16) |
| Quantized size | 82.8 GB (16 shards) |
| Quant format | NVFP4 via nvidia-modelopt 0.43.0, group_size=16, quant_algo: W4A16_NVFP4 |
| Quantized modules | 37,008 Linear layers: per layer 256 experts × {gate,up,down}_proj + shared_expert.{gate,up,down}_proj |
| Kept BF16 | self_attn*, linear_attn* (Gated DeltaNet), mlp.gate (router), mlp.shared_expert_gate, embed_tokens, lm_head, all norms, vision tower (333 tensors), MTP head (785 tensors) |
| Calibration | 64 × 1024 tokens of public text (CNN/DailyMail, Wikipedia EN/DE, Python source, UltraChat); only used for the shipped input_scale (see below) |
| Conversion date | 2026-11-16 |
Quality
No KLD against BF16 yet. The BF16 model (250 GB) does not fit on one GB10, so the end-to-end comparison used for my other releases could not be run. What was measured instead:
Per-layer and per-expert quantization error
Recorded during quantization with --layer-stats: relative L2 error of the output against BF16, each layer on its BF16 inputs, 64 samples × 1024 tokens.
Table with columns: median, max | median | max |
|---|
| whole layer (residual removed) | 0.080 | 0.112 (layer 40, 1.4×) |
| MLP block | 0.112 | 0.124 (layer 16, 1.1×) |
| shared expert | 0.096 | 0.147 (layer 17, 1.5×) |
| routed experts (12,134 with ≥ 32 tokens) | 0.138 | 0.342 (layer 0, 2.5×) |
- No layer stands out. The error per layer and per MLP block is flat across the depth of the model.
- A few experts in layers 0 and 1 do. 6 of the 7 experts above 1.5× the median sit in the first two layers (worst: layer 0 expert 237 at 0.34, layer 0 expert 48 at 0.30, layer 1 expert 99 at 0.24); all of them are routed often (1,500–3,800 tokens). From layer 2 on, the worst expert of every layer stays below 1.5× its layer median, with one exception (layer 29 expert 50, 1.65×, 257 tokens). Their effect on the layer output is small — layer 0 has the lowest layer error of the whole model (0.0075) — but its end-to-end cost was not measured.
- For comparison, the same measurement on Qwen3.6-35B-A3B found no expert above 1.3× the median.
Dead experts in the base model
13 experts in layers 3–13 have weights that are already ≈ 0 in the BF16 checkpoint (max |w| between 1e-14 and 1e-33), yet the router sends them thousands of tokens. They quantize to a valid (tiny) scale — no NaN/Inf, no zero scale anywhere in this repo — and vLLM serves them correctly. This is a property of the base model, not of the quantization.
Smoke tests
On a single DGX Spark (GB10), vLLM 0.31.0, thinking off, temperature 0: coherent German explanation of why the sky is blue, correct arithmetic, working Python code, in-character roleplay and English creative writing.
Speed (single DGX Spark, GB10, vLLM 0.31.0, MTP k=1)
Table | |
|---|
| Prefill, ~4k-token prompt | 2,139 tok/s |
| Decode, 1 stream | 21.7 tok/s |
Decode, 4 streams (total, --max-num-seqs 4) | 60.8 tok/s |
| MTP draft acceptance | 77 % (mean acceptance length 1.77) |
GEMM path: MarlinNvFp4LinearKernel.
Memory: the model takes 77.3 GiB (incl. MTP). At --gpu-memory-utilization 0.85 with --max-num-seqs 1, vLLM reported 6.9–9.1 GiB for KV cache across loads, depending on what else holds memory on the Spark. The full 262K context needs 7.02 GiB and failed once at 6.9 GiB, so the command below uses 192K (196608), which fits with margin. Only 12 of 48 layers (plus MTP) carry a KV cache. That leaves little room for anything else on the Spark.
Usage
vLLM (serve, single DGX Spark)
vllm serve Kaleto/Qwen3.5-122B-A10B-NVFP4 \ --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --attention-backend flashinfer \ --max-model-len 196608 \ --max-num-seqs 1 \ --gpu-memory-utilization 0.85 \ --enable-prefix-caching
- No
--quantization flag needed; vLLM reads the modelopt config.
--gpu-memory-utilization is a fraction of total memory. On a 128 GB Spark, 0.85 is needed for a usable KV cache; at 0.8 only 3.6 GiB (66K tokens) are left. For the full --max-model-len 262144, run nothing else on the Spark (it loaded at 0.85 with 9.1 GiB KV cache); each +0.01 adds ~1.2 GiB of headroom (untested above 0.85).
- Drop
--speculative-config to serve without MTP; the MTP tensors are then ignored.
- Text only: add
--language-model-only to skip the vision encoder and free memory for KV cache.
W4A4 instead: the calibrated input_scale tensors are included. Set quant_algo to NVFP4 in config.json and hf_quant_config.json to get the W4A4 path. On Qwen3.6-35B-A3B that cost ~45 % more KLD for no decode gain; it was not measured on this model.
Sampling
Qwen's recommendations (all with min_p=0, repetition_penalty=1.0):
Table with columns: Mode, temperature, top_p, top_k, presence_penalty| Mode | temperature | top_p | top_k | presence_penalty |
|---|
| Thinking, general | 1.0 | 0.95 | 20 | 1.5 |
| Thinking, precise coding | 0.6 | 0.95 | 20 | 0.0 |
| Non-thinking, general | 0.7 | 0.8 | 20 | 1.5 |
| Non-thinking, reasoning | 1.0 | 1.0 | 40 | 2.0 |
Toggle thinking per request with chat_template_kwargs: {"enable_thinking": false}. min_p and logit_bias have no effect while speculative decoding is active (vLLM limitation).
How it was made
distrib-nvfp4, sequential mode: one decoder layer at a time is loaded in BF16, quantized with NVFP4_DEFAULT_CFG, calibrated on the hidden states of the previous layer and exported. Peak memory is about one layer plus the hidden states, so a single Spark handles the 250 GB source. During calibration every token is routed to all 256 experts, so every expert gets real statistics. The vision tower and MTP head are copied verbatim from the source checkpoint.
The W4A16 weights do not depend on the calibration data (weight scales come from the weights alone); calibration only sets the input_scale used by the optional W4A4 path. All 37,008 quantized modules carry a calibrated input_scale.
Files
model-000{01..16}-of-00016.safetensors — NVFP4 expert weights + scales, BF16 for everything else, vision tower and MTP head
model.safetensors.index.json — 149,741 tensors
config.json — upstream config + quantization_config (W4A16_NVFP4, 147 ignore entries)
hf_quant_config.json, generation_config.json
layer_stats.json — the per-layer / per-expert error statistics summarized above (experts: [rel. error, tokens]; dead experts show NaN, unrouted ones null)
- tokenizer, chat template and processor files — unchanged from upstream
Acknowledgments
- Qwen team for Qwen3.5-122B-A10B
- NVIDIA for DGX Spark / GB10, NVFP4 and modelopt
- vLLM project for modelopt NVFP4, Marlin FP4 and Qwen3.5 MTP support
License
Apache 2.0, inherited from Qwen/Qwen3.5-122B-A10B.