Why this quant
- 📦 8.6% smaller than nvidia's own NVFP4 and ~7.7% faster. 19.2 GiB against 21.0, and
2,208–2,218 tok/s against 2,054 at concurrency 32, with accuracy indistinguishable from it.
That is the whole reason this repo exists; on the two columns that reproduce, it wins both.
- 🗜️ 3.3× smaller than BF16, 2.7× its throughput. 19.2 GiB against 62.9, 2,208 tok/s against 824.
- 🎯 Accuracy is a tie with everything, BF16 included. Pooled over both suites the field spans
86.6–87.7, and repeat runs of this one checkpoint span 86.6 to 87.1. No accuracy claim is made in
either direction.
- 🛠️ Tool calling is this model's strong axis. 82.4 on a 200-item tool-calling suite, the best of
any model we have quantized, and 1.9 points ahead of its own BF16.
- 🎚️ This is the smallest legal expert group on this architecture.
moe_intermediate is 1856, so
group 128 does not divide down_proj (1856/128 = 14.5) and cannot be built. Group 64 is the floor,
which is why this build sits where it does rather than lower.
- 🔧 No calibration data, and none needed. Weights-only round-to-nearest, which matters here
only because the build it beats on size and speed is a calibrated one.
- 🚀 The MTP head is preserved, so nvidia's speculative-decoding drafters remain usable.
- 🧩 Stock vLLM, plain
compressed-tensors. No patched runtime, no custom kernels.
Serve it
vllm serve primitive-ai/Nemotron-3.5-Lightning-30B-A3B-mixed-INT4-INT8 --max-model-len 32768
Measured
1,370 items across fourteen public benchmarks. A 1,170-item knowledge suite and a 200-item
tool-calling suite, under one fixed protocol for every row: temperature 0.6 / top_p 0.95 /
top_k 20, thinking forced on, a 16,384-token budget, concurrency 32 plus a 60-item single-stream
subset, all rows back to back in one sitting on one RTX PRO 6000 Blackwell.
Table with columns: build, bpw, size, overall, knowledge, call, abstain, runs k/a, finished, out/answer, tok/s @ 32, per-token latency| build | bpw | size | overall | knowledge | call | abstain | runs k/a | finished | out/answer | tok/s @ 32 | per-token latency |
|---|
| BF16 (reference) | 16.000 | 62.9 G | 87.0 | 88.1 | 82.5 | 72.5 | 1/1 |
overall is one number over both suites: the 1,170 knowledge and 200 tool-calling items pooled
as 1,370, weighted 85.4% and 14.6% by item count. Repeat runs of one checkpoint moved it by
about a point, so treat a gap below 1.0 as a tie.
Table with columns: build, agentic, call, abstain| build | agentic | call | abstain |
|---|
| BF16 (reference) | 80.5 | 82.5 | 72.5 |
| nvidia NVFP4 | 84.0 | 85.0 | 80.0 |
| a 5.24 bpw variant of ours | 82.7 | 85.8 | 70.0 |
| this repo | 82.4 |
call is accuracy on the 160 rows that require a call; abstain is the 40 whose correct action is
to call nothing. Benchmarks: BFCL v4 (30, including irrelevance), xLAM/APIGen (45), ToolACE
(45), Glaive v2 (40), nvidia When2Call (40, the abstention rows). Tool schemas go in the system prompt
and the model answers with a JSON array of calls, the same way for every model. An item counts only
if every expected call is present with acceptable arguments and no call is invented.
Abstention is the weak axis for every model we have measured (52–82%), so a build can look strong
on overall and still over-call.
Why there is no accuracy claim on this card. The last row is a drift control: the same checkpoint
re-measured later, at 86.6 against a three-run mean of 87.1. Across the whole table overall
spans 86.6–87.3 including BF16, so that column is one tie. Throughput reproduced to 0.7% between
those two runs (2,204 → 2,219) and nvidia's row to 0.1% across two sittings, which is why every claim
above is a size or speed claim.
Correction (26 Aug 2026): the 5.24 bpw variant is a tie, not an upgrade. An earlier version of
this card read its single 85.5 on tool calling as a 3.5-point lead and told readers whose workload is
tool calling to take the larger build. Under repeats:
Table with columns: agentic, knowledge, runs a/k | agentic | knowledge | runs a/k |
|---|
| 5.24 bpw variant | 82.67 | 87.99 | 3/2 |
| this repo (5.01 bpw) | 82.4 | 87.86 | 5/3 |
This checkpoint's own five agentic runs span 80.5 to 85.0 — it drew an 85.0 itself. The
"3.5-point gap" was one draw against another from inside a single build's range. Take the smaller
one: 1.8 GiB less, 4% faster, level on both halves.
Worth noting separately: this model holds 77.5% on the abstention rows, against 52–62% for most
of the field, the one model we have measured that is good at not calling.
A negative result worth having. Going up in bits does not help here. A 5.93 bpw build of this
model costs 2 GiB and ~10% throughput against this one and buys nothing that survives the noise
floor. On a model that is 89% routed experts, more bits are not what accuracy is short of, and
the reason this build does not go lower is arithmetic, not choice: moe_intermediate is 1856, so
group 128 does not divide down_proj and cannot be built.
Comparable with our other models
Accuracy numbers move for reasons that have nothing to do with the model: a shorter token budget, a
different temperature, or whether the model was allowed to reason at all. So every number in this
table, on this card and on our other cards, comes from one fixed protocol.
The same 1,370 items: a 1,170-item knowledge suite (MMLU-Pro, ARC-Challenge, HellaSwag, WinoGrande,
CommonsenseQA, BoolQ, OpenBookQA, GSM8K, MATH-500) and a 200-item tool-calling suite (BFCL v4,
xLAM/APIGen, ToolACE, Glaive v2, nvidia When2Call). temperature 0.6, top_p 0.95, top_k 20,
thinking forced on, a 16,384-token budget, no reasoning parser, scoring the last ANSWER: in the
reply. Concurrency 32 on one RTX PRO 6000 Blackwell, each model's rows in one sitting. Auto-scored,
no LLM judge. Both halves are means of at least three runs per build.
Table with columns: model, shape, size, overall, knowledge, call, abstain, finished, out, tok/s @ 32| model | shape | size | overall | knowledge | call | abstain | finished | out | tok/s @ 32 |
|---|
| Laguna-XS-2.1 | 31 B MoE | 19.3 GiB | 81.7 | 83.8 | 68.4 | 73.5 | 98.9% |
Read overall with finished. overall scores an answer that overran the token budget as wrong,
but it cannot say whether the model needed the room or failed to stop; finished and out separate
those. A gap under 1.0 is a tie. The tok/s column comes from each model's own sitting and drifts
a few percent between sittings, so read it as a bracket.
call and abstain are the tool-calling suite's two halves, reported separately. call is
accuracy on the 160 items that require a tool call; abstain is the 40 whose correct action is to
call nothing. They used to be pooled into one agentic number, and the pooling misled: a model with
ordinary call accuracy and unusual abstention discipline outscored models that are better at
actually making calls. Weight them by your own workload's mix.
Table with columns: benchmark, Laguna-XS-2.1, Nemotron-3.5-Lightning-30B-A3B, Ornith-1.5-35B-A3B, Muse-Glimmer-30B, Qwen3.8-27B, Laguna-S-2.1, Qwen3.8-Flash-Next| benchmark | Laguna-XS-2.1 | Nemotron-3.5-Lightning-30B-A3B | Ornith-1.5-35B-A3B | Muse-Glimmer-30B | Qwen3.8-27B | Laguna-S-2.1 | Qwen3.8-Flash-Next |
|---|
| knowledge | | | | | | | |
| mmlu_pro | 79.0 | 82.0 |
What's quantized to what
Table with columns: tensors, format| tensors | format |
|---|
| routed experts and the shared expert (29.8 B params, 89% of the model) | INT4 (group 64) |
Mamba in_proj/out_proj, attention q/k/v/o_proj, lm_head | INT8 (channel) |
MTP head, embeddings, router gate, conv1d, norms | BF16 |
Weights-only round-to-nearest, no calibration data required or embedded. The shared expert shares the
routed experts' scheme because vLLM builds it inside the fused MoE on this architecture. It cannot be
given its own precision.