Why this quant
- 🎯 7.9× less drift than NVIDIA's 24GB build. +9.6×10⁻³ nats per token, against
+75.7 for NVIDIA's 2-bit release
and +13.7 for the community NVFP4 build. Top-1 agreement 96.5%, against
92.7% and 96.1%.
- ⚡ 607 tok/s at concurrency 32, against 440 for the 24GB build. The community
NVFP4 runs 613 with an FP8 KV cache. Single-stream the 24GB build is faster:
125.5 tok/s against 112.7 here, because it reads fewer bytes per token.
- 📏 Accuracy level with the reference. Knowledge 90.6 and tool calling 82.0,
against 90.5 and 82.0 for the reference and 89.7 and
81.0 for the 24GB build, which also finishes fewer answers inside the budget
(97.1% against 98.5%).
- 🗜️ 46.70 GB, against 24.08 GB for the 24GB build, 47.78 GB for the
community NVFP4 and 150 GB in BF16. On one 96 GB card the command below leaves a 4,074,466-token KV pool at the full
262,144-token window.
- 🧩 The image and kernels NVIDIA's 24GB build needs: the vLLM nightly it names, plus
b12x from PyPI.
Serve it
hf download primitive-ai/Nemotron-3.5-Bolt-74B-A9B-mixed-NVFP4-FP8 --local-dir ./Nemotron-3.5-Bolt-74B-A9B-mixed-NVFP4-FP8 docker run --gpus all --ipc=host -p 8000:8000 -v $PWD:/models --entrypoint /bin/bash \ vllm/vllm-openai:nightly-77154ca6116ae34cbfe314beef1016bde9b63989 -c ' pip install b12x && VLLM_USE_V2_MODEL_RUNNER=1 vllm serve /models/Nemotron-3.5-Bolt-74B-A9B-mixed-NVFP4-FP8 \ --trust-remote-code --max-model-len 262144 --gpu-memory-utilization 0.92 \ --moe-backend b12x \ --attention-backend FLASHINFER --mamba-backend flashinfer --mamba-cache-mode align \ --limit-mm-per-prompt "{\"image\":1,\"video\":0}" \ --enable-auto-tool-choice --tool-call-parser qwen3_coder \ --reasoning-parser-plugin /models/Nemotron-3.5-Bolt-74B-A9B-mixed-NVFP4-FP8/ultra_v3_reasoning_parser.py --reasoning-parser ultra_v3'
Checked with exactly this command on an idle card. Boot takes 270 s including the pip install (the weights load in
34 s, 43.96 GiB) and the KV pool is 4,074,466 tokens. What is 17*23? comes back as 391 with the reasoning split
off by the parser, a weather question comes back as the parsed call get_weather {"city": "Lisbon"}, and an image of
the number 427 is read as 427.
With NVIDIA's LiLiCorr drafter (the --speculative-config from NVIDIA's recipe, 7 tokens), single-stream decode on 24 real prompts goes from 129 to 302 tok/s on this build and from 163 to 377 on the 24GB build (median per request).
Measured
One RTX PRO 6000 Blackwell, 96 GB, one card, no tensor parallelism. The 1,170-item knowledge suite and the 200-item
tool-calling suite, temperature 0.6 / top_p 0.95 / top_k 20, thinking on, a 16,384-token budget, concurrency
32, auto-scored with no LLM judge. Throughput is 8K in / 512 out, prefix-cache free, two seeds per cell, every build
benched on the same card one after another.
Table with columns: build, format, size, drift (×10⁻³ nats/token), top-1 agreement, knowledge, tool-calling, call, abstain, finished, tok/s @1, tok/s @32| build | format | size | drift (×10⁻³ nats/token) | top-1 agreement | knowledge | tool-calling | call | abstain | finished | tok/s @1 | tok/s @32 |
|---|
| 8-bit reference (ours, not published) | Q8_0, 16-bit activations | 80.92 GB | 0 (reference) | 100% | 90.5 | 82.0 |
Drift and agreement come from 400 traces of the knowledge suite (292,873 completion tokens), scored
by each build in one pass; the standard error over traces is 0.4 for this build. Tool calling is
three runs per build, knowledge two. KV cache is BF16 for every row except †: its config turns on FP8 KV and vLLM
follows it.
The 24GB build ran without its speculative drafter; with NVIDIA's FP8 KV setting it measures 127.9 and
442 tok/s.
What this table does not cover:
- Images and video. The suites are text only; the image check above is one prompt.
- Context past 32K. The model's window is 262,144 tokens; we did not test the 4-bit and 8-bit layers there.
- Other GPUs. Only the RTX PRO 6000 Blackwell (sm_120) was tested.
Comparable with our other models
Accuracy numbers move for reasons that have nothing to do with the model: a shorter token budget, a
different temperature, or whether the model was allowed to reason at all. So every number in this
table, on this card and on our other cards, comes from the one fixed protocol described above, the
same 1,370 items, auto-scored, no LLM judge.
Table with columns: model, shape, size, overall, knowledge, call, abstain, finished, out/answer| model | shape | size | overall | knowledge | call | abstain | finished | out/answer |
|---|
| Laguna-XS-2.1 | 31 B MoE | 19.3 GiB | 81.7 | 83.8 | 68.4 | 73.5 | 98.9% | 1097 |
|
overall pools the two suites as 1,370 items, weighted 85.4% knowledge and 14.6% tool calling by
item count. Read it with finished: overall scores an answer that overran the token budget as
wrong, and cannot say whether the model needed the room or failed to stop. A gap under 1.0 in
overall is a tie. Sizes are as each card reports them, which mixes GB and GiB.
What's quantized to what
Nemotron-3.5-Bolt-74B-A9B is 74.9 B parameters including the vision tower, 9 B active per token; 88% of them are
routed experts.
Table with columns: params, share | params | share |
|---|
| routed experts, 40 MoE layers, 240 to 496 experts each, up + down | 66.14 B | 88.3% |
| Mamba-2 in_proj + out_proj, 40 layers | 4.38 B | 5.9% |
| shared experts, 40 layers | 1.76 B | 2.4% |
| embeddings and LM head (untied) | 1.07 B | 1.4% |
| vision tower and projector | 0.84 B | 1.1% |
| MoE latent projections | 0.34 B |
Table with columns: modules, count, format| modules | count | format |
|---|
| routed experts, up_proj + down_proj | 27,648 | NVFP4 weights (E2M1, E4M3 scale per 16, FP32 global), 16-bit activations |
| shared experts, up_proj + down_proj | 80 | NVFP4, 16-bit activations |
| Mamba in_proj + out_proj | 80 | FP8 E4M3, one scale per output channel; activations FP8, one scale per token |
| MoE latent fc1 / fc2 | 80 | Q8_0 (int8, FP16 scale per 32), 16-bit activations |
| attention q / k / v / o | 32 | Q8_0 |
| LM head |
ModelOpt MIXED_PRECISION (quantized_layers in config.json and hf_quant_config.json). Q8_0 uses the
packing: ggml block layout that NVIDIA's 24GB build uses for the same modules.
Licence
OpenMDW License Agreement 1.1, the base model's licence; the full text is in this repository, with NVIDIA's
bias, explainability, privacy and safety notes from the original release. This is a modified version of
nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16: the weights listed above were quantized and config.json carries a
quantization_config. Everything else is unchanged. The model and its copyright belong to NVIDIA.