What is quantized
Table with columns: Part, Precision, Why| Part | Precision | Why |
|---|
Routed experts up_proj / down_proj (40 MoE layers × 512 experts) | int4, group 64, symmetric | ~90% of the bytes; served by the fused MoE WNA16 kernel |
Shared experts up_proj / down_proj | int4, group 64, symmetric | the maximum quantization the card can serve |
Latent projections fc1_latent_proj / fc2_latent_proj | int4, group 64, symmetric | as above |
Attention q_proj / k_proj / v_proj / o_proj (8 layers) | int4, group 64, symmetric | as above |
Mamba-2 mixers (in_proj, out_proj, conv1d, A_log, D, dt_bias, norm) | bf16 | int4 Mamba-2 projections gave fluent, wrong answers in a community build of the same family |
MoE routers (gate, e_score_correction_bias), norms, embeddings, lm_head | bf16 | standard |
| MTP head, vision tower (RADIO) and projector | bf16 | not quantized; this build is served text-only |
- 41,152 quantized modules; 79.8 GB in 64 shards (the bf16 source is 248.9 GB).
- Group 64 is the largest group size that divides every quantized input dimension after the TP=2 split. Each expert's
2,688-wide intermediate becomes 1,344 per card, which 128 does not divide.
- The expert target also carries the name
None: vLLM v0.32.0 checks the schemes of experts.0.up_proj, experts.0.down_proj and
experts.0.None (the absent up projection of a non-gated expert), and refuses the layer when one of them reads unquantized.
The name does not exist in the checkpoint, so matching it is harmless. The published config.json has it.
- The
gate_proj name is part of the expert target although these experts are not gated (relu², up/down only). vLLM's
compressed-tensors MoE loader checks all three projection names and refuses a layer whose schemes differ.
How it was made
- Tool: llm-compressor 0.14.0
model_free_ptq with compressed-tensors 0.19.0, data-free (no calibration set), on CPU only.
- Scheme:
num_bits=4, symmetric=true, strategy=group, group_size=64, observer memoryless-mse; no activation
quantization.
- Targets (both the checkpoint's and vLLM's module names):
re:^language_model\.(?:model|backbone)\.layers\.\d+\.mixer\.(?:experts\.\d+\.(?:gate_proj|up_proj|down_proj|None)|shared_experts\.(?:up_proj|down_proj)|fc[12]_latent_proj|[qkvo]_proj)$
- Source revision:
fb0ee0bc6b3f70342afd8d84cb80dbadbaa8a1fe.
- Built in 43 min with 5 worker processes × 8 threads (two shards per batch, 32 batches). The output is deterministic: a rerun
gives byte-identical shards.
- Verified after the build: every scheme field, the targets in both name forms, every shard present, and a dequantization
check of one shard (relative error 0.097–0.098 on the routed experts, the normal level for int4).
Serving (vLLM, ROCm)
Tested with vLLM v0.32.0 (vllm/vllm-openai-rocm:v0.32.0, ROCm 7.2) and v0.33.0 (vllm/vllm:v0.33.0-rocm, ROCm 10.0) on
2 × MI210. Quantization is detected from the config; do not
pass --quantization.
vllm serve <this repo> \ --tensor-parallel-size=2 --trust-remote-code \ --max-model-len=32768 --gpu-memory-utilization=0.92 --kv-cache-dtype=auto \ --max-num-batched-tokens=2048 --language-model-only \ --limit-mm-per-prompt='{"image":0,"video":0,"audio":0}' \ --reasoning-parser=nemotron_v3 --tool-call-parser=qwen3_coder
- The routed experts run on
CompressedTensorsWNA16MoEMethod (fused MoE, Triton WNA16 kernel). The int4 shared experts,
latent projections and attention run on TritonW4A16LinearKernel, which is slower than bf16 on gfx90a. That is the price of
the smaller footprint.
- A fused-MoE kernel config tuned for this shape on the MI210
(
E=512,N=1344,device_name=Instinct_MI210,dtype=int4_w4a16.json) ships in the serving stack that built this model. On
gfx90a, int4 MoE tiles with BLOCK_SIZE_K wider than the group size (here 128 or 256) fault the GPU at this shape, so the
tuned config only uses BLOCK_SIZE_K of 32 or 64.
Results
Measured on 2 × MI210 at TP=2 with the tuned fused-MoE config, on vLLM v0.32.0 unless noted:
Table with columns: Check, Result| Check | Result |
|---|
| Weights per card | 34.12 GiB |
KV cache capacity (--gpu-memory-utilization=0.92, bf16 KV) | 1,698,650 tokens; 51.8x concurrency at 32,768 tokens per request |
| KV cache capacity on vLLM v0.33.0, ROCm 10.0 | 2,383,607 tokens (+40%: ROCm 10 halves the CUDA-graph memory pool) |
| Output tokens per second, 1,536 prompt / 128 output tokens | 25.3 at concurrency 1, 38.6-40.0 at 2, 60.7-65.0 at 4 |
| Time to first token | 0.57-0.58 s at concurrency 1, 1.2-1.3 s at 4 |
| Correctness checks (temperature 0, thinking on and off) | 17 x 60 = 1,020; train trip of 155 min; a get_weather tool call all passed |
| lm-evaluation-harness, 50 samples per task, thinking on | gsm8k 0.98, humaneval 1.00, mbpp 0.96 |
The task scores are a 50-sample sanity check that the quantization kept the model's quality, not a full benchmark; compare
them with the original model card's numbers only loosely. Throughput varies by about 8% from run to run.
License and attribution
This is a derivative of NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 by NVIDIA. Use of this model is governed by the same
license as the original: the OpenMDW License Agreement, version 1.1, whose full text is in
this repository's LICENSE file. The custom modeling and processing code (*.py) is copied unchanged from the original
repository, with its notices.
The quantization changes the numerical precision of the weights listed above and nothing else. It is not an official NVIDIA
release, and NVIDIA has not reviewed or endorsed it. For the model's intended use, safety, bias and explainability details,
see the original model card. Its guidance on
ethical considerations and safety guardrails applies to this build unchanged.