Why this quant
- 🗜️ 2.5× smaller than BF16, 28% smaller than Qwen's own FP8. 20.7 GiB against
51.7 GiB BF16 and 28.7 GiB FP8.
- 🎯 Accuracy ties BF16 and eight of the nine builds measured. 88.8 vs 89.3 pooled over 1,370
items. Repeating one unchanged checkpoint moves that number by ~1.1 points, which is wider than
the whole field's spread.
- ⚡ 2.0× throughput at concurrency 32, 2.5× at concurrency 1 versus BF16.
- 💸 At the cheap end of the field on output tokens. 651 per answer against a 625–818 spread,
at a 99.7% completion rate. Both go straight to your serving bill.
- 🖥️ Verified end to end on Ampere as well as Blackwell. Native FP4/FP8 kernels on Blackwell,
Marlin W·A16 fallback on A100, where the BF16
lm_head is what makes that path load.
- 🚀 Speculative decoding survives quantization. The in-checkpoint MTP head is
preserved: up to +101% with a DSpark drafter.
- 🔧 No calibration data, no custom runtime. Weights-only round-to-nearest,
compressed-tensors, stock vllm serve.
Serve it
vllm serve primitive-ai/Qwen3.8-27B-mixed-NVFP4-FP8 \
--reasoning-parser qwen3 \
--speculative-config '{"method":"mtp","num_speculative_tokens":2}'
Measured
1,370 items across fourteen public benchmarks. A 1,170-item knowledge suite and a 200-item
tool-calling suite, under one fixed protocol for every row: temperature 0.6 / top_p 0.95 /
top_k 20, thinking forced on, a 16,384-token budget, concurrency 32, all rows back to back in one
sitting on one RTX PRO 6000 Blackwell.
Table with columns: build, size, overall, knowledge, call, abstain, runs k/a, finished, out/answer, tok/s @ 32, per-token latency, tok/s @ 1| build | size | overall | knowledge | call | abstain | runs k/a | finished | out/answer | tok/s @ 32 | per-token latency | tok/s @ 1 |
|---|
| Qwen BF16 | 51.7 G | 89.3 | 91.0 | 85.6 | 55.0 |
overall is one number over both suites: the 1,170 knowledge and 200 tool-calling items pooled
as 1,370, weighted 85.4% and 14.6% by item count. Repeat runs of one checkpoint moved it by
about a point, so treat a gap below 1.0 as a tie.
call is accuracy on the 160 rows that require a call; abstain is the 40 whose correct action is
to call nothing. Benchmarks: BFCL v4 (30, including irrelevance), xLAM/APIGen (45), ToolACE
(45), Glaive v2 (40), nvidia When2Call (40, the abstention rows). Tool schemas go in the system prompt
and the model answers with a JSON array of calls, the same way for every model. An item counts only
if every expected call is present with acceptable arguments and no call is invented.
Abstention is the weak axis for every model we have measured (52–82%), so a build can look strong
on overall and still over-call.
† These three checkpoints ship a kv_cache_scheme that enables an FP8 KV cache automatically.
That is a real memory advantage, and it also lifts their throughput against builds running a BF16 KV
cache, so the speed columns are not purely a comparison of weights.
Read the runs column before the ordering. This build's figures are means of twelve knowledge
runs; every community build except RadixArk is a single draw. Re-running RadixArk interleaved with
this build on one host settled its 92.0 to 91.3, and left the two 0.35 apart on knowledge. Since
one unchanged checkpoint moves overall by ~1.1 points, the 1.3 points separating this whole field
is not a ranking. Decide on size, output tokens and hardware instead.
A100 40 GB. Everything runs Marlin W·A16 there, and a build whose lm_head is quantized will
not load on that path at all, which is why this one leaves it at BF16:
Table with columns: build, acc, tok/s @ conc 32, tok/s @ conc 1| build | acc | tok/s @ conc 32 | tok/s @ conc 1 |
|---|
| this repo | 88.1 | 790.0 | 53.9 |
Repeat measurements of this exact checkpoint span 89.7 to 91.0 on the knowledge suite across
twelve runs in five sittings, sd 0.39, and its tool-calling half moves about a point over five.
Throughput moved −4.5% between two of those sittings, which is why every row of a table here is
measured back to back rather than assembled from separate runs.
Speculative decoding
A separate measurement run from the table above. The baselines differ, so the
two are never plotted together.
Table with columns: speculative config, conc 1, conc 16| speculative config | conc 1 | conc 16 |
|---|
| none | 59.4 tok/s | 396.6 tok/s |
MTP ({"method":"mtp","num_speculative_tokens":2}) | +59% | +61% |
DSpark drafter (RadixArk/Qwen3.8-27B-DSpark, n=4) | +101% | +83% |
Gotcha: the RadixArk drafter's config.json needs
architectures: ["Qwen3DSparkModel"]. As shipped it says DSparkDraftModel,
which vLLM routes to the DeepSeek-V4 class and crashes on hc_mult.
Reasoning effort is a real cost lever
Qwen3.8's chat template exposes reasoning_effort, and its default is xhigh. Measured on this
checkpoint, same 1,170 items, same sitting:
Table with columns: reasoning_effort, acc, finished, out/answer, tok/s @ conc 32, per-token latencyreasoning_effort | acc | finished | out/answer | tok/s @ conc 32 | per-token latency |
|---|
xhigh (the default) | 90.5 | 99.7% | 613 | 894 | 35.8 ms |
medium | 89.7 | 99.8% | 587 | 901 |
curl localhost:8000/v1/chat/completions -d '{
"model": "...", "messages": [...],
"chat_template_kwargs": {"reasoning_effort": "low"}
}'
low costs 1.2 accuracy points and returns about 25% cheaper answers. 14% fewer output tokens
and 14% lower per-token latency, which compound. On a throughput-bound deployment where a point of
accuracy is worth less than a quarter of the serving bill, that is the setting to run.
medium is not worth using. It saves 4% of tokens for 0.8 points, the worst trade of the three.
Comparable with our other models
Accuracy numbers move for reasons that have nothing to do with the model: a shorter token budget, a
different temperature, or whether the model was allowed to reason at all. So every number in this
table, on this card and on our other cards, comes from one fixed protocol.
The same 1,370 items: a 1,170-item knowledge suite (MMLU-Pro, ARC-Challenge, HellaSwag, WinoGrande,
CommonsenseQA, BoolQ, OpenBookQA, GSM8K, MATH-500) and a 200-item tool-calling suite (BFCL v4,
xLAM/APIGen, ToolACE, Glaive v2, nvidia When2Call). temperature 0.6, top_p 0.95, top_k 20,
thinking forced on, a 16,384-token budget, no reasoning parser, scoring the last ANSWER: in the
reply. Concurrency 32 on one RTX PRO 6000 Blackwell, each model's rows in one sitting. Auto-scored,
no LLM judge. Both halves are means of at least three runs per build.
Table with columns: model, shape, size, overall, knowledge, call, abstain, finished, out, tok/s @ 32| model | shape | size | overall | knowledge | call | abstain | finished | out | tok/s @ 32 |
|---|
| Laguna-XS-2.1 | 31 B MoE | 19.3 GiB | 81.7 | 83.8 | 68.4 | 73.5 | 98.9% |
Read overall with finished. overall scores an answer that overran the token budget as wrong,
but it cannot say whether the model needed the room or failed to stop; finished and out separate
those. A gap under 1.0 is a tie. The tok/s column comes from each model's own sitting and drifts
a few percent between sittings, so read it as a bracket.
call and abstain are the tool-calling suite's two halves, reported separately. call is
accuracy on the 160 items that require a tool call; abstain is the 40 whose correct action is to
call nothing. They used to be pooled into one agentic number, and the pooling misled: a model with
ordinary call accuracy and unusual abstention discipline outscored models that are better at
actually making calls. Weight them by your own workload's mix.
Table with columns: benchmark, Laguna-XS-2.1, Nemotron-3.5-Lightning-30B-A3B, Ornith-1.5-35B-A3B, Muse-Glimmer-30B, Qwen3.8-27B, Laguna-S-2.1, Qwen3.8-Flash-Next| benchmark | Laguna-XS-2.1 | Nemotron-3.5-Lightning-30B-A3B | Ornith-1.5-35B-A3B | Muse-Glimmer-30B | Qwen3.8-27B | Laguna-S-2.1 | Qwen3.8-Flash-Next |
|---|
| knowledge | | | | | | | |
| mmlu_pro | 79.0 | 82.0 |
What's quantized to what
Table with columns: tensors, format| tensors | format |
|---|
all 64 layers' MLP gate/up/down_proj; linear-attn in_proj_qkv/in_proj_z on layers 8–63 | NVFP4 (group 16) |
full-attn q/k/v/o_proj; linear-attn in_proj_qkv/in_proj_z on layers 0–7; linear_attn.out_proj | FP8 E4M3 (channel) |
lm_head, linear-attn in_proj_a/in_proj_b gates, conv1d, norms, embeddings, vision tower, MTP head |
Weights-only round-to-nearest, no calibration.