At a glance
Table | |
|---|
| Base model | nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16 |
| Quantization | AutoRound W4A16, int4, group 128, symmetric, stored as GPTQ |
| Size | 41 GB (the BF16 source is 139.5 GiB) |
| Tested hardware | two Intel Arc Pro B70 (32 GB each) |
| Engine | vLLM 0.31.0 XPU, tensor parallel 2 |
| Decode, one request (C1), 886 prompt tokens | 60.5 tok/s |
| Prefill, one request (C1), 886 prompt tokens | 2,925 tok/s |
| Context | 65,536 default; 262,144 (the model maximum) loads and works |
The rest of this card covers what is quantized, how it was made, how to run it, and the numbers
measured on the two cards.
What is quantized
Table with columns: Part, Format| Part | Format |
|---|
Routed experts (27,648 Linear layers), shared experts, latent MoE projections, Mamba-2 in_proj/out_proj, attention q/k/v/o | int4, group 128, symmetric (27,920 Linear layers) |
Vision tower (RADIO), projector, embeddings, lm_head, MoE router gates, norms, Mamba conv1d | BF16, byte-identical to the source |
Router e_score_correction_bias | F32, byte-identical |
Mamba A_log, D, dt_bias | F32 holding exactly the BF16 source values |
Bolt has no MTP layer (num_nextn_predict_layers: 0). NVIDIA's speculative drafter for it is
LiLiCorr, a separate model.
How it was made
- AutoRound
main 08d7085e plus the Nemotron-H Omni support from
intel/auto-round#2512 (approved, not merged when this
was made). transformers 5.12.1.
scheme W4A16, group_size 128, sym, iters 200, nsamples 128, seqlen 1536, disk-streamed load,
low_gpu_mem_usage=True, both cards (device_map 0,1). 5 h 57 min on two B70s.
- Bolt's remote Mamba-2 code builds 6-D tensors in its plain PyTorch path and runs out of memory during
calibration. The export used an einsum version of four lines (same output, peak 25.8 GiB down to 3.0
GiB). The repo ships NVIDIA's original
modeling_nemotron_h.py.
The full process (timeline, every command, all 15 fixes, the checks and the scripts) is in the cookbook:
How the Bolt 74B AutoRound W4A16 quant was made and checked.
The ready-to-run recipe and launch script are in
VLLM-TP2-AUTOROUND-W4A16.md,
and the B70 Launcher has it as a one-click recipe.
Running it on two Intel Arc Pro B70 (vLLM 0.31.0 XPU)
Stock vLLM 0.31.0 needs three changes on this hardware. The files are in b70/:
- Per-worker GPU visibility and oneCCL settings for TP2 on boards without GPU peer-to-peer:
patch_vllm_worker_affinity.py
and the four CCL_SYCL_* variables below.
- Loopback for oneCCL when the container uses
--network host. Without it the two ranks paired
endpoints on different interfaces (wifi and a Docker bridge) and waited forever at startup.
- Top-k 9. Bolt routes each token to 9 experts. The XPU MoE kernel
remap_hidden_states
(vllm-xpu-kernels) only supports top-k 1, 2, 4, 6, 7, 8, 10, 12 and 16, and fails with
Unsupported TopK value. b70/patch_xpu_moe_topk_pad.py pads each token to 10 experts by repeating
the last one with router weight 0. The output is unchanged; it costs about 11% more expert work.
docker run -d --name bolt --device /dev/dri --group-add <render-gid> --ipc=host --shm-size 8g \ --cap-add SYS_PTRACE --network host \ -e ZE_AFFINITY_MASK=0,1 -e B70_WORKER_AFFINITY=1 -e \ -e VLLM_WORKER_MULTIPROC_METHOD=spawn \ -e CCL_SYCL_ALLREDUCE_SIMPLE_THRESHOLD=4294967296 \ -e CCL_SYCL_REDUCE_SCATTER_SIMPLE_THRESHOLD=4294967296 \ -e CCL_SYCL_ALLGATHERV_SIMPLE_THRESHOLD=4294967296 \ -e CCL_SYCL_ALLTOALL_TMP_BUF=1 \ -e CCL_KVS_IFACE=lo -e FI_TCP_IFACE=lo -e VLLM_HOST_IP=127.0.0.1 \ -v /path/to/this/repo:/model:ro \ -v /path/to/patch_vllm_worker_affinity.py:/patches/patch_vllm_worker_affinity.py:ro \ -v /path/to/this/repo/b70/patch_xpu_moe_topk_pad.py:/patches/patch_xpu_moe_topk_pad.py:ro \ -v /path/to/this/repo/b70/tp2-entry.sh:/tp2-entry.sh:ro \ --entrypoint bash \ vllm/vllm-openai-xpu@sha256:95ac815038c4b3537b173798bfe3b89bdf525a52bd653cd14d4deb01f1062be7 \ /tp2-entry.sh serve /model --served-model-name nemotron-bolt --host 127.0.0.1 --port 8000 \ --trust-remote-code --tensor-parallel-size 2 --dtype bfloat16 --kv-cache-dtype fp8 \ --attention-backend TRITON_ATTN --max-model-len 65536 --max-num-batched-tokens 8192 \ --block-size 64 --max-num-seqs 4 --gpu-memory-utilization 0.85 \ --limit-mm-per-prompt '{"image":1,"video":0}' --enable-prefix-caching --mamba-cache-mode align
Do not add --enforce-eager: with XPU graphs decode is 6x faster (59.5 vs 10.1 tok/s on an 886-token
prompt in an earlier 3-run screen). Expect "Using XPUExpertsWNA16" in the log and 20.0 GiB of weights
per card. At --gpu-memory-utilization 0.85 each card uses about 26.6 GiB (measured at 16,384 length).
For the full 262,144-token context, set --max-model-len 262144. It loads with room for 409,375 fp8 KV
tokens (1.56 full-length requests). It works, but reading a 250K-token prompt takes about 35 minutes,
which is why 65,536 is the default.
Results (two Arc Pro B70, 2026-10-16)
Greedy, thinking off, one request at a time, 128 output tokens. Every prompt starts with a unique random
prefix, so nothing comes from the prefix cache (cache hits during the run: 0). One warmup, then 5 runs per
size; the table shows the median and the range. Both cards capped at 180 W (measured average draw 171 W
and 172 W). Server started with --max-model-len 262144 and prefix caching on.
Table with columns: Prompt tokens, Prefill (tok/s), Time to first token, Decode (tok/s)| Prompt tokens | Prefill (tok/s) | Time to first token | Decode (tok/s) |
|---|
| 886 | 2,925 (2,895-2,943) | 0.30 s | 60.5 (60.2-60.7) |
| 8,192 | 2,014 (1,993-2,026) | 4.1 s | 50.7 (49.9-50.7) |
| 32,000 | 871 (835-873) | 36.7 s | 33.5 (32.0-33.5) |
| 64,000 | 486 (463-487) | 131.7 s | 23.0 (23.0-23.0) |
[!NOTE]
One machine, self-reported, not reproduced independently.
Prefill is prompt tokens divided by time to first token. The same host runs NVIDIA's 24 GB GGUF on
one card with llama.cpp at 28.2 tok/s decode.
Checks:
- Static check against the BF16 source: all 27,920 quantized layers present, no NaN, Inf or zero
scales, every unquantized tensor identical (or the same values in F32).
- Exact answers, 8/8: arithmetic,
sum(i*i for i in range(4)) = 14, a loop sum, division, facts,
letter counts, a number sequence.
- Six-question smoke, 5/6. The miss is vision with the question before the image: it answers only
"42" for a red circle with "42" in it. With the image first it answers "Red circle with 42". NVIDIA's
GGUF on llama.cpp answered fully in the question-first order. I have not separated whether the
quantization or vLLM's image placement causes this.
- Greedy output with XPU graphs is byte-identical to eager mode on all six smoke cases.
- GSM8K, first 200 questions of the test split, greedy, thinking off: 192/200 (96.0%).
- A separate battery of 6 synthetic images with its own wording: 6/6 with the image first and 6/6 with
the question first. It includes the red circle; with this wording the question-first answer was
"42 red circle", so the smoke miss above depends on the prompt.
- Multi-turn chats with the prefix cache on: 3/3. Cache on and off give identical answers (6/6).
Four requests at once give the same answers as one at a time (4/4).
- Finding one hidden fact in a long prompt: found at 31K tokens (10%, 50% and 90% depth), at 127K
tokens (same three depths) and at 249,212 tokens (50% depth).
[!WARNING]
Vision, question-before-image order: put the image before the text.
Not tested: NVIDIA's LiLiCorr speculative drafter, tool calling, video, more than 4 requests at once,
benchmarks other than the GSM8K sample above. These will be added here when measured.
License
Same as the base model: OpenMDW 1.1. This is a derivative of
NVIDIA's release; see the base model card
for its intended use, limitations and data.