What is quantized
The recipe copies the one NVIDIA used for Nemotron 3.5 Lightning NVFP4 (ModelOpt 0.44), with one difference: the Mamba in/out projections stay in BF16 here, where Lightning has them in FP8.
Table with columns: Layers, Format| Layers | Format |
|---|
| Routed experts (up and down projections, 27,648 layers) | NVFP4 W4A16 |
| Shared experts | NVFP4 W4A16 |
lm_head | NVFP4 W4A16 |
| Mamba in/out projections, attention, router gates, latent projections, conv1d, embeddings | BF16 |
| Vision encoder (C-RADIO) and projector | BF16 |
NVFP4 W4A16 stores weights as 4-bit E2M1 values with one FP8 E4M3 scale per block of 16 and one FP32 scale per tensor. Activations stay in BF16. The checkpoint is in ModelOpt's MIXED_PRECISION format, the same tensor layout as NVIDIA's own NVFP4 releases.
Weight-only quantization needs no calibration data. Each weight was rounded to nearest with ModelOpt's NVFP4QTensor.quantize, one shard at a time. The mean relative error of the dequantized weights against BF16 is 0.094. scripts/convert_nvfp4.py is the script that made this checkpoint.
Running it
Tested with vLLM nightly 0.31.1rc1.dev296+gfe4df3df0 (image vllm/vllm-openai:nightly-fe4df3df0eb703862c686b256e9fe44fa94c8945) on a DGX Spark. vLLM detects the format as modelopt_mixed without a --quantization flag and runs the FP4 layers on its Marlin W4A16 kernels.
export VLLM_USE_V2_MODEL_RUNNER=1vllm serve sapidlabs/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-NVFP4-W4A16 \ --trust-remote-code \ --max-model-len 65536 --max-num-seqs 8 --gpu-memory-utilization 0.80 \ --enable-chunked-prefill \ --attention-backend FLASHINFER --mamba-backend flashinfer --mamba-cache-mode align \ --enable-auto-tool-choice --tool-call-parser qwen3_coder \ --reasoning-parser-plugin ultra_v3_reasoning_parser.py --reasoning-parser ultra_v3
ultra_v3_reasoning_parser.py is in this repo. Turn thinking off per request with "chat_template_kwargs": {"enable_thinking": false}. scripts/serve_bolt_nvfp4.sh is the Docker version of this command.
Measured on one DGX Spark (GB10, 128 GB)
Table | |
|---|
| Memory for weights | 48.35 GiB |
KV cache at --gpu-memory-utilization 0.80 | 43.98 GiB, 3,805,271 tokens |
| Weight load | 5 min 40 s |
| Decode, one request | 17 tok/s, thinking on or off |
| Decode, 8 requests at once | 84 tok/s total |
The Mamba in/out projections are still BF16: 4.4B parameters, about 9 GB that are read for every token.
Quality
GSM8K, the first 200 test questions, thinking off, temperature 0: 194/200 (97.0%). The prompt asked for step-by-step working and a final Answer: <number> line.
Also checked: plain answers with thinking off, a coding task with thinking on (a working nth_prime and the correct value of nth_prime(100)), and a tool call parsed by the qwen3_coder parser. Image and video input have not been tested.
License
The base model is released by NVIDIA under the OpenMDW License Agreement, version 1.1. This checkpoint is distributed under the same license, and LICENSE holds its text. NVIDIA's original model card is kept unchanged as ORIGINAL_MODEL_CARD.md.