Quantization
Table with columns: Part, Format| Part | Format |
|---|
| Language model linear layers | NVFP4 weights and activations, group size 16 |
| KV cache | FP8 (e4m3) |
Vision encoder (model.visual*) | bf16 |
lm_head, embeddings | bf16 |
Gated DeltaNet conv1d, in_proj_a, in_proj_b | bf16 |
MTP head (mtp.*) | bf16 |
Calibration: 512 samples of 512 tokens from the Model Optimizer default mix (abisee/cnn_dailymail and nvidia/Nemotron-Post-Training-Dataset-v2), max calibration.
MTP head
The base model was saved through transformers, which does not load the Qwen3.5 MTP head, so its checkpoint has no mtp.* tensors. This repo adds the 15 mtp.* tensors from Qwen/Qwen3.8-27B unchanged, in bf16, in model-mtp.safetensors. mtp* is in the quantization ignore lists, so vLLM loads the head unquantized.
The MTP head comes from the original model, not from the modified one. Speculative decoding verifies every draft token against the main model, so the head changes speed only, not output.
Usage with vLLM
vLLM reads the modelopt_fp4 quantization and the FP8 KV cache from the checkpoint, so no --quantization flag is necessary.
vllm serve jbowman512/Qwen3.8-27B-unsycophantic-NVFP4 \ --kv-cache-dtype fp8 \ --reasoning-parser qwen3 \ --speculative-config '{"method":"mtp","num_speculative_tokens":5}'
If vLLM compiles the FlashInfer FP4 GEMM kernel at first start (a vLLM install without the flashinfer-jit-cache package), set MAX_JOBS=4 in the environment. Without a limit, the parallel CUTLASS compile can run out of host RAM.
Testing
Tested on one RTX PRO 6000 Blackwell (96 GB) with vLLM 0.32.0:
- The model loads in 19.5 GiB of GPU memory.
- Chat answers to factual, coding, and engineering prompts were correct and coherent.
- The model kept the base model's refusal to agree with a false premise.
- With MTP and 5 speculative tokens, 41% of draft tokens were accepted (about 3 tokens for each decode step).
No benchmark comparison against the bf16 model was run. NVIDIA lists NVFP4 as validated for Qwen3.5 MoE models, not for the dense Qwen3.5 vision-language model. Image input was not tested.
Reproduce
pip install "nvidia-modelopt[hf]==0.48.0" "transformers==5.18.0"git clone --depth 1 --branch 0.48.0 https://github.com/NVIDIA/Model-Optimizer.gitpython Model-Optimizer/examples/hf_ptq/hf_ptq.py \ --pyt_ckpt_path jbowman512/Qwen3.8-27B-unsycophantic \ --qformat nvfp4 --kv_cache_qformat fp8_cast \ --calib_size 512 --calib_seq 512 --batch_size 4 \ --export_path Qwen3.8-27B-unsycophantic-NVFP4
Then add the MTP head from Qwen/Qwen3.8-27B as described above. The default calibration set is gated: request access to nvidia/Nemotron-Post-Training-Dataset-v2 first.