What is quantized
Table with columns: Part, Precision, Why| Part | Precision | Why |
|---|
Routed experts gate_proj / up_proj / down_proj (40 MoE layers × 512 experts) | int4, group 128, symmetric | almost all of the bytes; served by the fused MoE WNA16 kernel |
| Mamba-2 mixers (40 layers), shared experts, attention (8 layers), latent projections | bf16 | int4 Mamba-2 projections gave fluent, wrong answers in a community build of this model |
MoE routers, norms, embeddings, lm_head, vision tower | bf16 | standard; this build is served text-only |
- 27,648 quantized expert modules; 51.6 GB (the bf16 source is about 150 GB).
- This is a "puzzle" model: its layers are not identical, so after the TP=2 split the fused-MoE kernel sees 12 distinct shapes
(see below), which matters for tuning.
- Group 128 divides every quantized input dimension at tensor-parallel size 1 and 2.
How it was made
- Tool: llm-compressor 0.14.0
model_free_ptq with compressed-tensors 0.19.0, data-free (no calibration set), on CPU only
(4 worker processes, 51 min).
- Scheme:
num_bits=4, symmetric=true, strategy=group, group_size=128, observer memoryless-mse; no activation quantization.
- Targets (both the checkpoint's and vLLM's module names):
re:^language_model\.(?:model|backbone)\.layers\.\d+\.mixer\.experts\.\d+\.(?:gate_proj|up_proj|down_proj|None)$
(the name None stands for the absent up projection that vLLM probes on non-gated experts; it does not exist in the checkpoint).
- Source revision:
e09a4cedb1787db1760375ee5539f222cae0c672.
- Verified after the build: every scheme field, the targets in both name forms, every shard present.
Serving (vLLM, ROCm)
Tested with vLLM v0.33.0 (vllm/vllm:v0.33.0-rocm, ROCm 10.0) on 2 × MI210. Quantization is detected from the config; do not
pass --quantization.
vllm serve <this repo> \ --tensor-parallel-size=2 --trust-remote-code \ --max-model-len=32768 --gpu-memory-utilization=0.92 --kv-cache-dtype=auto \ --max-num-batched-tokens=2048 --language-model-only \ --limit-mm-per-prompt='{"image":0,"video":0,"audio":0}' \ --reasoning-parser=nemotron_v3 --tool-call-parser=qwen3_coder
The routed experts run on CompressedTensorsWNA16MoEMethod (fused MoE, Triton WNA16 kernel).
Tuned fused-MoE configs (optional)
moe-configs/ holds 12 Triton fused-MoE kernel configs, one per shape (E = experts per layer, N = expert intermediate width per
card at tensor-parallel size 2), tuned for this build on two MI210 with vLLM v0.33.0 (Triton 3.8.0):
Table with columns: E, N| E | N |
|---|
| 240 | 768, 1024, 1152 |
| 304 | 1024, 1152, 1280 |
| 368 | 1024, 1152, 1280 |
| 432 | 1152, 1280 |
| 496 | 1280 |
vLLM does not read them from the model repository: download the folder and point VLLM_TUNED_CONFIG_FOLDER at it. The startup log
then names each file (Using configuration from ...) instead of Using default MoE config.
hf download aidevco/Nemotron-3.5-Bolt-74B-A9B-W4A16-g128 --include 'moe-configs/*' --local-dir .export VLLM_TUNED_CONFIG_FOLDER=$PWD/moe-configs
- Each shape was tuned on its own (all 12 batch sizes from 1 to 2048, both cards on one chunk queue, finalists re-timed 10 times),
with the
BLOCK_SIZE_K tiles capped at the int4 group size (128).
- Serving A/B (default, tuned, tuned, default; fresh pod per run, 1536 prompt and 128 output tokens, concurrency 1, 2, 4): no measurable
difference. Output tokens per second changed by +1.2% at concurrency 1 and by 0.0% to +0.8% at 2 and 4 on a warm prefix cache; the one tuned
run that dipped (cold start, concurrency 2 and 4) was matched by the other. The experts are a small share of a step in this Mamba-2 hybrid. The
configs are harmless and published for completeness; no speed-up is claimed.
- Only valid for these exact shapes, this card and dtype. ROCm 7.2 images name the card
Instinct_MI210: copy each file under that name
too. A newer vLLM or Triton can change the kernel; re-check before relying on them.
Results
Measured on 2 × MI210 (64 GB), vLLM v0.33.0, TP=2, --max-model-len=32768, --gpu-memory-utilization=0.92, default MoE config:
Table with columns: Quantity, Value| Quantity | Value |
|---|
| Weights per card | 23.89 GiB |
| KV cache | 3,683,757 tokens (112x concurrency at 32,768 tokens) |
| Output tokens/s, concurrency 1 | 36.6 |
| Output tokens/s, concurrency 2 | 58.5 |
| Output tokens/s, concurrency 4 | 85.4 |
| Time to first token (mean) | 400 ms (c1), 550 ms (c2), 885 ms (c4), 1536-token prompts |
Four correctness checks passed at temperature 0 (17 times 60 = 1,020; a train-time word problem with thinking off and on, 155 minutes, no repetition loop;
a get_weather tool call with the right argument); the community int4-Mamba-2 build failed them.
No task-accuracy scores are claimed until they are measured.
License and attribution
This is a derivative of NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16 by NVIDIA. Use of this model is governed by the same license as the
original: the OpenMDW License Agreement, version 1.1, whose full text is in this repository's
LICENSE file (copied unchanged from the original). The custom modeling and processing code (*.py) is copied unchanged from the
original repository, with its notices.
The quantization changes the numerical precision of the weights listed above and nothing else. It is not an official NVIDIA release,
and NVIDIA has not reviewed or endorsed it. For the model's intended use, safety, bias and explainability details, see the
original model card; its guidance applies to this build unchanged.