Why this quant
- 🔁 The release's weights, bit for bit. All 302,795,194,368 routed-expert values reproduce the
released MXFP4 values exactly. Each E8M0 scale over 32 weights becomes two E4M3 scales over 16
against a power-of-two global, and no value needed rounding.
- ⚡ The experts run on native FP4. vLLM serves them with FlashInfer CUTLASS W4A4 on sm_120. Over
the full knowledge suite at concurrency 16 that is 532 tok/s against the release's 402.
- 🎯 Tool-calling level with the release. 78.6 over five runs, the release also 78.6. Knowledge
91.1 against 90.3.
- 📏 Activation scales set per layer. FP4 activations need one global scale per layer. The last
MoE layer's expert intermediate reaches about 60,000 on the prompts we measured, 22× past what a
scale of 1.0 can hold, so a single constant would clip it.
- 🧮 The cost is KV cache. NVFP4 stores 4.5 bits per weight to MXFP4's 4.25, so this build is 9.48 GB
larger than the release. On two 96 GB cards that leaves a 250,094-token pool, against 627,110 for
the release.
Serve it
hf download primitive-ai/MiMo-V2.6-Flash-RL-NVFP4 --local-dir ./MiMo-V2.6-Flash-RL-NVFP4 SP=/usr/local/lib/python3.12/dist-packages/vllmdocker run --gpus all --ipc=host --shm-size 32g -p 8000:8000 -v $PWD:/models \ -v $PWD/MiMo-V2.6-Flash-RL-NVFP4/vllm_patch/mimo_v2.py:$SP/model_executor/models/mimo_v2.py:ro \ vllm/vllm-openai:mimo-v26-x86_64-cu130 \ --model /models/MiMo-V2.6-Flash-RL-NVFP4 \ --tensor-parallel-size 2 --trust-remote-code --generation-config vllm \ --max-model-len 131072 --gpu-memory-utilization 0.95 \ --limit-mm-per-prompt '{"image":4,"video":0,"audio":0}' \ --reasoning-parser mimo --tool-call-parser mimo --enable-auto-tool-choice
The day-0 image (vLLM 0.29.1rc1.dev449) loads ModelOpt mixed-precision checkpoints, but its MiMo
loader assumes the release's own FP8 layout: the fused qkv loader writes to weight_scale_inv,
while ModelOpt's block-FP8 layers name that parameter weight_scale and store it 4-D.
vllm_patch/mimo_v2.py is the image's file with those spots bridged; the 21-line diff is
vllm_patch/mimo_v2.diff. Without the mount, loading stops with a KeyError.
Setting audio to 0 skips loading the audio encoder. The DFlash drafter ships as released, but
speculative decoding was not measured with this build.
Measured
Two RTX PRO 6000 Blackwell, 96 GB each, tensor parallel 2. The 1,170-item knowledge suite and the
200-item tool-calling suite, temperature 0.6 / top_p 0.95 / top_k 20, thinking on, a
16,384-token budget, concurrency 16, auto-scored with no LLM judge. bf16 KV, no speculative decoding,
same box, same day.
Table with columns: build, size, knowledge, tool-calling, call, abstain, finished, tok/s @1, tok/s @16| build | size | knowledge | tool-calling | call | abstain | finished | tok/s @1 | tok/s @16 |
|---|
| release, MXFP4 experts | 172.92 GB | 90.3 | 78.6 | 82.8 | 62.0 | 99.0% | 112.2 | 402 |
Tool-calling is the mean of five runs per build, with a within-build standard deviation of 0.8 to
0.9, so the column is one band. Knowledge is one run per quantized build and two for the
release; gaps under 1.2 are ties. tok/s @1 is 60 real prompts one at a time; tok/s @16 is the
knowledge suite's own aggregate.
Comparable with our other models
Accuracy numbers move for reasons that have nothing to do with the model: a shorter token budget, a
different temperature, or whether the model was allowed to reason at all. So every number in this
table, on this card and on our other cards, comes from the one fixed protocol described above, the
same 1,370 items, auto-scored, no LLM judge.
Table with columns: model, shape, size, overall, knowledge, call, abstain, finished, out/answer| model | shape | size | overall | knowledge | call | abstain | finished | out/answer |
|---|
| Laguna-XS-2.1 | 31 B MoE | 19.3 GiB | 81.7 | 83.8 | 68.4 | 73.5 | 98.9% | 1097 |
|
overall pools the two suites as 1,370 items, weighted 85.4% knowledge and 14.6% tool calling by
item count. Read it with finished: overall scores an answer that overran the token budget as
wrong, and cannot say whether the model needed the room or failed to stop. A gap under 1.0 in
overall is a tie. Sizes are as each card reports them, which mixes GB and GiB.
What's quantized to what
MiMo-V2.6-Flash-RL is 309 B parameters, 15 B active, and 98% of those parameters are routed experts.
Table with columns: tensors, count, format| tensors | count | format |
|---|
| routed experts, layers 1 to 47 | 36,096 projections | NVFP4: E2M1 codes, E4M3 scale per 16, FP32 global, W4A4 |
qkv_proj ×48, layer-0 dense MLP, MTP layers | 63 modules | FP8 E4M3, 128×128 blocks, byte-identical to the release |
o_proj, embeddings, lm_head, routers, vision and audio encoders | the rest | BF16, byte-identical to the release |
ModelOpt MIXED_PRECISION, one quantized_layers entry per module; the dense FP8 entries are tagged
FP8_PB_WO, the name vLLM's ModelOpt linear table accepts for block FP8. gate and up of each expert
share one global scale, so the fused projection vLLM builds keeps the exact values. vLLM reduces
expert input scales to one per layer, and the checkpoint stores them that way.
The expert weights are not re-quantized. The activation scales are the only new numbers in this
checkpoint.