Why this quant
- 🔬 The six FP8 layers were measured, not guessed. Every layer was scored by running the real
model on tool-call prompts with all experts at NVFP4, upgrading one layer at a time to FP8, and
seeing how far the final hidden state moved back toward BF16. Layers 7, 13, 18, 23, 29 and 35 won.
- 📉 Closest to the original of the three quantized builds. Those six layers remove 19.7% of the
representation drift that 4-bit quantization introduces, for six layers of forty.
- 🎯 Accuracy is level with BF16, like every build here. 90.6 knowledge and 76.5 tool-calling
against the original's 89.6 and 75.3.
- ⚖️ Between the endpoints on both axes. 26.04 GB against 23.91 and 38.13; 147 tok/s single-stream
and 990 at concurrency 32, in both cases between the 4-bit and 8-bit builds.
- 🧩 Stock vLLM, no patches. vLLM resolves the compressed-tensors scheme per MoE layer and runs
both kernels in one model: FlashInfer CUTLASS for the NVFP4 layers, Triton FP8 for the other six.
Serve it
hf download primitive-ai/Nex-N2.5-mini-mixed-NVFP4-FP8 --local-dir ./Nex-N2.5-mini-mixed-NVFP4-FP8
docker run --gpus all --ipc=host -p 8000:8000 -v $PWD:/models \
vllm/vllm-openai:v0.28.0 \
--model /models/Nex-N2.5-mini-mixed-NVFP4-FP8 \
--max-model-len 32768 --gpu-memory-utilization 0.92 \
--enable-auto-tool-choice --tool-call-parser qwen3_coder \
--reasoning-parser qwen3
Qwen3_5MoeForConditionalGeneration is in the 0.28.0 release, so no nightly is needed. The numbers
on this page were measured on 0.28.1rc1.dev472. We booted this build on 0.28.0 as well: 181 s to
healthy, both MoE kernels selected as expected, generation sane.
Two things about this model's chat template are worth knowing before you wire it up. It emits an
empty thinking block by default, so replies arrive with two leading newlines and the reasoning
field holds only whitespace; a short correct answer looks like "\n\n4". And reasoning_effort
(low / medium / xhigh) is a per-request field here, unlike some other reasoning models where
it is fixed at server start.
Measured
One RTX PRO 6000 Blackwell, 96 GB, one card. The 1,170-item knowledge suite and the 200-item
tool-calling suite, temperature 0.6 / top_p 0.95 / top_k 20, thinking on, a 16,384-token
budget, concurrency 32, auto-scored with no LLM judge. Throughput is 8K in / 512 out, prefix-cache
free, two seeds per cell. Every row below was run on the same box in the same sitting.
Table with columns: build, size, knowledge, tool-calling, call, abstain, finished, tok/s @1, tok/s @32| build | size | knowledge | tool-calling | call | abstain | finished | tok/s @1 | tok/s @32 |
|---|
| BF16 original | 70.24 GB | 89.6 | 75.3 | 79.7 | 58.7 | 99.7% | 141.0 | 664 |
| FP8 experts |
Tool-calling is the mean of two runs per build (three for BF16). Pooled within-build spread is
0.60 standard deviations over those nine runs, and the four build means span 1.7, so the column is
one band. Read the ordering with that in mind: it runs opposite to precision, which no mechanism
would predict, and is the clearest evidence that this suite cannot resolve the difference between
4-bit and 16-bit experts on this model.
Speed does separate, and it crosses over. FP8 experts decode 13% faster single-stream because the
4-bit path pays for activation quantization and group scales on every token, a cost that does not
amortize at batch 1. At concurrency 32 the ranking inverts and NVFP4 wins by 13%, because there
the bottleneck is weight bandwidth and 4-bit moves half the bytes. Pick by the concurrency you
actually run.
Comparable with our other models
Accuracy numbers move for reasons that have nothing to do with the model: a shorter token budget, a
different temperature, or whether the model was allowed to reason at all. So every number in this
table, on this card and on our other cards, comes from the one fixed protocol described above, the
same 1,370 items, auto-scored, no LLM judge.
Table with columns: model, shape, size, overall, knowledge, call, abstain, finished, out/answer| model | shape | size | overall | knowledge | call | abstain | finished | out/answer |
|---|
| Laguna-XS-2.1 | 31 B MoE | 19.3 GiB | 81.7 | 83.8 | 68.4 | 73.5 | 98.9% | 1097 |
|
overall pools the two suites as 1,370 items, weighted 85.4% knowledge and 14.6% tool calling by
item count. Read it with finished: overall scores an answer that overran the token budget as
wrong, and cannot say whether the model needed the room or failed to stop. A gap under 1.0 in
overall is a tie. Sizes are as each card reports them, which mixes GB and GiB.
How the six layers were chosen
A layer's own quantization error turns out to be a poor guide. Measured in isolation, every layer of
this model is distorted by about the same fraction at 4 bits: 2.39% on average, 1.48% to 2.64% across
all 40. Nothing in that spread tells you which layers to protect.
So the selection was made on the model instead of on the tensors. With all 40 layers at NVFP4, one
layer at a time was upgraded to FP8 inside the running engine, and the final hidden state was compared
against the BF16 original over 128 tool-call prompts:
gain(l) = || all-NVFP4 - BF16 || - || all-NVFP4 except l at FP8 - BF16 ||
Layers 7, 13, 18, 23, 29 and 35 came out on top, and together they remove 19.7% of the drift that
4-bit quantization introduces. The measurement is deterministic: three re-runs on identical weights
moved the result by 0.00000, and two independent passes agreed to the last decimal.
Two results from that sweep are worth stating because they contradict the obvious approach:
- Upgrading a layer can make the model worse. On 25 of the 40 layers, moving that layer from
NVFP4 to FP8 pushed the final hidden state further from BF16, by as much as 19% of the total gap.
Quantization errors partially cancel across layers, and improving one layer in isolation can break
a cancellation. Any selection method that scores layers independently is blind to this.
- Local error picks the wrong layers. Ranking by each layer's own block-output error selects the
top of the stack (32, 35 through 39), which by the end-to-end measure removes only 5.5% of the gap,
a third of what the layers above achieve for the same six layers. Calibrating that local criterion on
tool-call prompts alone rather than a general mix reproduced the same six layers exactly, so the
problem is the criterion, not the calibration data.
What the allocation does not do is move the benchmark. All four builds score inside one band, and
this card's table says so.
What's quantized to what
Nex-N2.5-mini is 35.11 B parameters, and they are distributed very unevenly:
Table with columns: params, share | params | share |
|---|
| routed experts, 40 layers × 256 × (gate+up+down) | 32.21 B | 91.8% |
| linear attention (Gated DeltaNet), 30 layers | 1.01 B | 2.9% |
embed_tokens and lm_head, untied, vocab 248320 | 1.02 B | 2.9% |
| vision tower, 27 blocks | 0.45 B | 1.3% |
| full attention, 10 layers | 0.27 B | 0.8% |
Table with columns: tensors, count, format| tensors | count | format |
|---|
| routed experts, 34 layers | 26,112 modules | NVFP4, group 16 |
| routed experts, layers 7 / 13 / 18 / 23 / 29 / 35 | 4,608 modules | FP8 E4M3, per output channel |
| everything else | 946 tensors | BF16, byte-identical to the source |
compressed-tensors, format mixed-precision, two config groups whose targets name specific layers.
vLLM resolves the scheme per MoE layer, so interleaving is fine, but all three projections of one
layer must agree, which is why the unit of allocation is a whole layer. The source ships experts as two batched 3-D tensors per layer;
they are unfolded into per-expert modules, which is the layout vLLM's loader consumes. gate and up
share one weight_global_scale per expert. vLLM fuses those two halves and keeps a single scale,
warning and taking the maximum when they disagree, which silently shrinks half the weights, so the
build is audited for that before release: 20,480 fused groups checked, no mismatches.
Weights-only round-to-nearest, no calibration. Nothing in the quantization is fitted to data.