Why this quant
- 🗜️ 3.0× smaller than BF16, 39% smaller than the official FP8. 22.6 GiB against 67.0 GiB and
36.7 GiB.
- 🎯 The only quant of this model measured tied with BF16 on knowledge. 91.5 (91.9 on a repeat)
against BF16's 92.0 on a 1,170-item suite — inside the ~1-point run-to-run spread. The official
NVFP4 scores 90.9 and the official FP8 91.2.
- 🛠️ Tool calling is this build's weaker axis — and there is a sibling build for that. 71.8 on a
200-item tool-calling suite, the mean of six runs, against BF16's 75.1 (mean of four): real but
marginal at t=2.3, and not statistically distinguishable from either official quant. Pooled over
both suites this build scores 88.6, ahead of the official NVFP4 (88.3) and behind the FP8 (89.0).
If tool calling is your workload, take
our agentic build —
same size, 75.1 on that suite, level with BF16.
- ⚡ 1.72× BF16 throughput at concurrency 32, 1.22× single-stream — 1,454 tok/s against 845.
- 👁️ The vision tower is untouched. Vision, the projector and norms stay BF16: a VLM loses its
visual grounding before its words.
- 🚀 The MTP block is preserved, so speculative decoding remains available — the tooling drops it
by default, and dropping it is silent.
- 🖥️ BF16
lm_head. It costs a little size and speed, and it is what keeps the Ampere
(Marlin W·A16) path loadable; a quantized head crashed that path on a sibling architecture.
- 🔧 Stock vLLM,
compressed-tensors, no patches.
Serve it
vllm serve primitive-ai/Ornith-1.5-35B-A3B-mixed-NVFP4-FP8 --max-model-len 32768
Text and image inputs both work. For long-context serving, --kv-cache-dtype fp8 buys a large
increase in KV capacity at negligible throughput cost; it is left off here so the numbers below
describe the checkpoint rather than a serving flag.
Measured
1,370 items across fourteen public benchmarks — a 1,170-item knowledge suite and a 200-item
tool-calling suite — under one fixed protocol for every row: temperature 0.6 / top_p 0.95 /
top_k 20, thinking forced on, a 16,384-token budget, concurrency 32 plus a 60-item single-stream
subset, all rows back to back in one sitting on one RTX PRO 6000 Blackwell.
Table with columns: build, size, overall, knowledge, agentic, finished, out/answer, tok/s @ 32, per-token latency| build | size | overall | knowledge | agentic | finished | out/answer | tok/s @ 32 | per-token latency |
|---|
| BF16 (reference) | 67.0 G | 89.5 | 92.0 | 75.1 | 99.2% | 708 | 845 | 37.9 ms |
| ornith-ai FP8 |
overall is one number over both suites — the 1,170 knowledge items and the 200
tool-calling items pooled as 1,370, which weights them 85.4% and 14.6% by item count. It is
the score you would get by running both files as one, not an average of two percentages. Repeat runs
of an identical checkpoint moved under a point, so read a gap below 1.0 in overall as a tie.
Table with columns: build, agentic, call, abstain| build | agentic | call | abstain |
|---|
| BF16 (reference) | 75.1 | 80.6 | 62.5 |
| ornith-ai FP8 | 74.9 | 78.8 | 65.0 |
| this repo | 71.8 | 73.1 | 52.5 |
| ornith-ai NVFP4 |
call is accuracy on the 160 rows that require a call; abstain is the 40 rows whose correct
action is to call nothing. Five public benchmarks: BFCL v4 (30, including irrelevance),
xLAM/APIGen (45), ToolACE (45), Glaive v2 (40) and nvidia When2Call (40, the abstention rows). Tool
schemas are supplied in the system prompt and the model answers with a JSON array of calls,
identically for every model — measured within 1.5 points of native tools= calling on our own
Qwen3.8 quant. An item counts only if every expected call is present with acceptable arguments and
no call is invented.
Abstention is the weak axis for every model we have measured (52–82%), so a build can look
strong on overall and still over-call.
Pooling weights tool calling at 14.6%, which is what item count says — and it shrinks a
tool-calling-only divergence to a seventh of its size. That is exactly why knowledge and agentic
stay beside overall in the table above instead of being folded away into it.
The last row is a drift control: the same checkpoint re-measured after two hours of other work came
back within 1.3% on throughput and 0.4 points on knowledge.
Every agentic figure above is a mean of repeated runs, and on this model that is essential.
Quantized builds of Ornith carry two to three times BF16's run-to-run spread on the 200-item
tool-calling suite: six runs of this checkpoint span 67.5 to 75.5 (sd 3.1), against BF16's 1.4
and the official quants' 1.2-1.7. An earlier version of this card reported a single run (69.0) as
though it were the number, and drew a conclusion from it that the data does not support. Corrected:
Table with columns: build, tool calling, n, sd| build | tool calling | n | sd |
|---|
| BF16 | 75.1 | 4 | 1.4 |
| ornith-ai FP8 | 74.9 | 4 | 1.7 |
| ornith-ai NVFP4 | 73.1 | 4 | 1.2 |
| this repo | 71.8 | 6 | 3.1 |
This build is 3.4 points below BF16 on tool calling (t=2.3, so real but marginal) and not
statistically distinguishable from either official quant -- the NVFP4 by 1.4 points at t=1.0. On the
pooled score it leads that NVFP4 (88.6 vs 88.3). Read differences under about 2 points here as noise.
🛠️ If tool calling is your workload, use
our agentic build
instead. Same size and format, calibrated for tool use: 75.1 on this suite -- level with BF16 --
with a third of this build's run-to-run spread, at the cost of 0.8 points of general knowledge.
Read this honestly: the official NVFP4 build is faster and smaller than this one — 15.6% more
aggregate throughput, 27% more single-stream, and 0.75 GiB less on disk. It gets there partly by
quantizing the 508 M-parameter lm_head, which is hit on every decoded token; this build keeps that
head at BF16. So the trade is roughly: their build if you want maximum speed on Blackwell, this one
if you want the closest accuracy to BF16 and an output head that keeps the Ampere fallback path
loadable. All four accuracy figures sit within about a point of each other, which is this suite's
run-to-run spread — treat that column as a tie and choose on the others.
Comparable with our other models
Accuracy numbers move for reasons that have nothing to do with the model — a shorter token budget, a
different temperature, or whether the model was allowed to reason at all. So every number in this
table, on this card and on our other model cards, comes from one fixed protocol:
- the same 1,370 items: a 1,170-item knowledge suite over nine public benchmarks (MMLU-Pro,
ARC-Challenge, HellaSwag, WinoGrande, CommonsenseQA, BoolQ, OpenBookQA, GSM8K, MATH-500) and a
200-item tool-calling suite over five more (BFCL v4, xLAM/APIGen, ToolACE, Glaive v2, nvidia
When2Call) — auto-scored, no LLM judge, no partial credit
overall is those two pooled by item count, 85.4% knowledge and 14.6% tool calling, which is
the score you would get by running both files as one. knowledge and agentic are its halves,
printed because a model can be strong on one and weak on the other and the pooled column alone
would not tell you- both halves are means of repeated runs, not single draws. Every build here was measured at
least three times per half. The agentic column needs it most — repeat runs of one checkpoint on
those 200 items move by as little as 0.8 points on some models and as much as 3.1 on others — but
the knowledge half moves too, by 0.1 to 0.4 points, and four of these builds were originally
published from one knowledge run each. Re-measuring changed the ranking: two builds that looked
0.2 apart are now tied
- one fixed decode setting for every model —
temperature 0.6, top_p 0.95, top_k 20, and
thinking forced on — so no model gets a number tuned in its own favour, and none is measured
with its reasoning switched off by a template default
Table with columns: model, shape, size, overall, knowledge, agentic, finished, out, tok/s @ 32| model | shape | size | overall | knowledge | agentic | finished | out | tok/s @ 32 |
|---|
| Laguna-XS-2.1 | 31 B MoE | 19.3 GiB | 81.7 | 83.8 | 69.6 | 98.9% | 1097 tok | 1523 |
|
Read overall and finished together. overall counts an answer that ran past the token budget
as wrong, which is fair — an answer you never receive is not an answer. But it cannot tell you why
one never arrived, and there are two very different reasons: a model that reasons at length and needs
the room, or a model that reached its answer and then failed to stop. finished is that missing
column and out is the per-reply cost, so the two together separate the cases.
A difference under 1.0 point in overall is inside the repeat spread and should be read as a
tie — pooling damps the noisy half, which is why the pooled column is the trustworthy one even where
the agentic column is volatile. The tok/s column comes from each model's own sitting; throughput on
this host drifts a few percent between sittings, so treat it as a bracket rather than a ranking.
Table with columns: benchmark, Laguna-XS-2.1, Nemotron-3.5-Lightning-30B-A3B, Ornith-1.5-35B-A3B, Muse-Glimmer-30B, Qwen3.8-27B, Laguna-S-2.1| benchmark | Laguna-XS-2.1 | Nemotron-3.5-Lightning-30B-A3B | Ornith-1.5-35B-A3B | Muse-Glimmer-30B | Qwen3.8-27B | Laguna-S-2.1 |
|---|
| knowledge | | | | | | |
| mmlu_pro | 79.0 | 82.0 | 89.5 | 89.0 | 89.5 |
What's quantized to what
Table with columns: tensors, format| tensors | format |
|---|
| routed expert projections (all 40 layers × 256 experts) | NVFP4 (group 16) |
full-attention q/k/v/o_proj; linear-attention in_proj_qkv/in_proj_z/out_proj; shared expert | FP8 E4M3 (channel) |
lm_head, embeddings, router, shared_expert_gate, the linear-attention state path (in_proj_a/in_proj_b/conv1d/A_log/dt_bias), norms, vision tower, MTP block |
Weights are quantized with GPTQ against a text-only calibration set; the vision tower is not
quantized, so image inputs are unaffected. Activations for the NVFP4 group are quantized dynamically
at runtime by vLLM.