JasonW2025

Laguna-S-2.1-PrismaQuant-gridbook-2.6bit-vllm

README

License: openmdw-1.1

What is quantized, and what deliberately is not

Table with columns: component, format, bpp, why
component	format	bpp	why
routed experts, layers 1–47 (`mlp.experts.{gate_up,down}_proj`)	NVFP4-CB two-tier v2, K16	2.28125	96 % of the parameters, ~3.9 % of them read per token
attention `q/k/v/o`, all 48 layers	compressed-tensors FP8 W8A8 dynamic	8	on the token path in full, every token
`mlp.shared_expert` (47 layers)	BF16	16	on the token path in full — see below
layer-0 dense MLP	BF16	16	on the token path in full
router, `e_score_correction_bias`, norms, `embed_tokens`, `lm_head`	BF16	16	tiny, or numerically load-bearing

The shared_expert is BF16 on purpose, and we have the measurement. We built the variant that codebooks it at the same 2.28 bpp as the routed experts. It saved 0.70 GiB of resident memory and cost 8 points on the agentic bench (75, 75 against this build's 83, 83 — both n=2 with zero within-arm spread, which is not typical of this bench and makes the difference hard to dismiss). It was also slower. A routed expert is read 10 times in 256; the shared expert is read every token. Quality cost scales with token exposure, and 0.70 GiB is not worth it.

The codebook fit is CB-GPTQ — columnwise OBS error propagation with codeword-batched assignment, damp=0.1, per-rung pooled Hessian-weighted codebooks, per-sub tables. Activations for the Hessians were captured from the real forward pass with the model on meta and one decoder layer's weights streamed in at a time, then validated by an identity rather than by shape (RMSNorm makes the per-token mean-square of X/w identically 1; the captured tensor measured 0.0084, a wrong tensor measures 36.55).

Serving

This artifact does not load on a stock pip install gridbook. It needs the plugin revision in serving/, which carries the Laguna load chain. Two of those patches are correctness, not performance:

config.py — _is_ignored matched by substring, so an ignore entry swallowed sibling modules whose names merely contained it; the module then built as an unquantized BF16 Linear and died on a missing scale key.
moe_toplevel_loader.py — fused native-NVFP4 expert stacks need a loader. Laguna's load_weights ends if name not in params_dict: continue, so an unhandled stack leaves modules at random init with no error at all.

serving/PINS.md lists the sha256 of every file. serving/40_serve_laguna.sh re-hashes them host-side before docker run and refuses to start on a mismatch — the pin is content-addressed rather than commit-addressed on purpose, because a commit sha says nothing about an uncommitted working-tree diff.

What serving/ is not. It is the plugin layer, not a complete runtime. It does not include the vLLM container image we serve inside, and the pins verify only the gridbook files — nothing about the vLLM build, PyTorch, CUDA or FlashInfer versions around them. We run a GB10 nightly build of vLLM; reproducing that is on you, and a different vLLM may need different patches than the ones here. Treat serving/ as necessary, not sufficient.

bash

vllm serve /model --served-model-name laguna-cb \  --quantization gridbook \  --max-model-len 32768 --max-num-seqs 4 \  --kv-cache-dtype auto \  --gpu-memory-utilization 0.90 \  --enforce-eager \  --no-enable-prefix-caching \  --trust-remote-code \  --enable-auto-tool-choice \  --tool-call-parser poolside_v1 --reasoning-parser poolside_v1 \  --default-chat-template-kwargs '{"reasoning_effort":"no_think"}'

--kv-cache-dtype auto (i.e. bf16) is deliberate. This artifact ships no calibrated KV scales (kv_cache_scheme: null, zero k_scale/v_scale tensors). Forcing fp8 would quantize the cache at scale 1.0 against a distribution nothing matched — measured at −2 points on this model for no speed gain. If you want FP8 KV, calibrate it first.

--enforce-eager is what we measured under; CUDA graphs are not a supported configuration for this build yet.

The reasoning parser is broken upstream, and the fix is in this repo

vLLM's PoolsideV1ReasoningParser derives from the DeepSeek-V3 parser, which selects its concrete implementation from chat_template_kwargs["thinking"] / ["enable_thinking"]. Both default to False, so it silently becomes an IdentityReasoningParser whose reasoning_start_str/reasoning_end_str are None — vLLM even logs "Auto-initialization of reasoning token IDs failed" and carries on. Chain-of-thought then lands in scored content and a stray </think> appears in roughly 68 of 69 bench scenarios.

serving/poolside_v1_reasoning_parser.py is a drop-in overlay (bind-mount it over the one in the image; no rebuild). It is worth 3 points: 83, 83 → 86, 86. This affects every Laguna deployment on affected vLLM builds, not just quantized ones.

Performance

Measured on one DGX Spark: GB10, sm_121a, 121.63 GiB unified memory, ~233 GB/s measured read bandwidth. Decode is marginal throughput — 60-token prompt, 256 completion tokens, median of 3 repeats.

Table with columns: this build, poolside/Laguna-S-2.1-NVFP4 (the base author's own)
	this build	`poolside/Laguna-S-2.1-NVFP4` (the base author's own)
size	35.03 GiB	67 GiB
decode	26.25 tok/s	19.6 tok/s
KV headroom @ util 0.90	1,084,424 tok	—
agentic bench, matched (no parser fix)	83, 83	87, 85

On the bench row: poolside's own release scores higher than this build. 87 and 85 against our 83 and 83, same harness, same day, same box, both without the parser overlay. (Their 19.6 tok/s is unaffected by the fp4 op — that release does not use this plugin.) Our 86, 86 comes from re-running this build with that overlay; poolside's NVFP4 has not been re-run with it, and since the overlay was worth +3 here it would very likely gain similarly. Comparing our 86 against their 87/85 would be comparing a fixed run against unfixed ones, so do not do it, and neither will we. The honest summary is: at the only matched comparison that exists, this build trades roughly 2–4 points of agentic score for half the size and 15 % more decode.

That trade is the point of the artifact. It is not a claim of parity.

Where the decode comes from. The fp4 activation-QDQ used to run as ~20 eager PyTorch operations per call (measured 82 µs at M=1, K=3072); serving/ replaces it with one CUDA kernel (8.2 µs), and this model makes 94 such calls per decoded token. That predicted 6.87 ms/token, i.e. 22.52 → 26.57 tok/s. Measured end-to-end: 26.25 tok/s — 1.2 % off the prediction, from a measured µs/call and a measured call count.

The serve log carries

text

[prismaquant-cb] act-qdq fp4=cuda (prismaquant::fp4_act_qdq, 1 launch/call)

. Check for that line. Its absence means the CUDA op was not found and the eager path is running: numerics are identical, throughput is not, so a benchmark without that line is measuring a different thing than this table.

Decode here is bandwidth-bound and follows ms/token ≈ active_bytes / 233 GB/s + ~19 ms, where active counts routed experts at 10/256. Resident bytes are the wrong denominator for a sparse MoE.

Quality — what the bench number is, and what it is not

These are agentic tool-use scores — 69 scenarios exercising tool selection, argument construction, multi-step planning, restraint and safety — run at temperature 0 with prefix caching off, twice per serve session. Every Laguna arm we measured, on one box, one harness:

Table with columns: arm, teb (n=2), parser overlay
arm	teb (n=2)	parser overlay
poolside official NVFP4 (67 GiB)	87, 85	no
ModelOpt NVFP4 W4A4 (65 GB)	88, 86	no
ModelOpt NVFP4 W4A16 (68.5 GB)	86, 85	no
this build (LAG1F, 35.03 GiB)	83, 83	no
this build + parser overlay	86, 86	yes
same, but shared_expert CB'd (LAG2)

Only the rows sharing a parser state are comparable. The overlay was worth +3 on this build and nothing else has been re-run with it.

It is not a fidelity measurement against the original weights, and this is not a nitpick:

No BF16 teacher exists for this model on this hardware. 117.6 B parameters in bf16 is 235 GB; a single Spark has 128. Nobody — us included — can currently report a KL or perplexity delta against the unquantized model at this scale on this box. Any claim that this quantization "matches" the original is unsupported, including ours.
Do not read small gaps as rankings. On this bench a byte-identical artifact re-benched in a fresh session has scored 5 points apart. In a controlled study on this stack we found the server is not deterministic at temperature 0 — the same request repeated returns different logits, and ~50 % of greedy replicates diverge within the first two tokens. Differences of a few points are not resolvable here. The shared-expert result quoted above is called out precisely because it is much larger than that floor.
Both runs scored 86 exactly; that is reassuring about the harness, not evidence of precision.
The 86 was measured with the plugin revision before the fp4_act_qdq CUDA op that ships in serving/. That op is bit-exact to the eager implementation it replaces — torch.equal, not a tolerance, across 50 checks spanning 12 shapes, four magnitude regimes and the exact-midpoint tie-break case — so we do not expect quality to move. But bit-exactness is an argument, not a re-measurement, and we have not re-run the bench with the op in the loop. Stated so you can weigh it yourself.

If you need a fidelity guarantee rather than a task score, measure it on your workload.

Known limitations

Single-GPU only as measured. TP/PP with this plugin is untested here.
GB10 / sm_121a is the only hardware this was measured on. The NVFP4-CB decode path wants NVFP4 tensor cores; off Blackwell the plugin falls back to a Triton path its own documentation calls "correct, not fast, and not a production serving target."
Long context is arithmetic, not measurement. The base model declares 1,048,576 tokens and the KV headroom above is real, but every bench here ran at ~316 tokens. No needle-retrieval or long-prefill test has been run.
No calibrated KV scales (see above).
config.json provenance was corrected on upload. The quantization_config.provenance block had carried the previous build's recipe/label — it described a BF16 hot path on an artifact whose hot path is FP8. That block is metadata only (the plugin pops it before use), so no numerics changed, but it was wrong and is now right.

Provenance and licensing

Base model: poolside/Laguna-S-2.1, OpenMDW-1.1 — LICENSE.md is included here as that license requires, along with the base model's own configuration_laguna.py and modeling_laguna.py (which config.json's auto_map references).
Quantizer: PrismaQuant (AURA-lineage Fisher/KL allocator), with our NVFP4-CB encoder.
Serving plugin: gridbook by RobTand, Apache-2.0 — serving/LICENSE.gridbook.Apache-2.0 is included. The files in serving/gridbook/ and serving/csrc/ are from that project; the load chain, per-stack rung dispatch, 2-D CB Linear lane, decode GEMV v2 and the fp4 activation-QDQ CUDA op are our changes.

Quantized by Jason Wong. No affiliation with poolside or with the gridbook author.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

JasonW2025

Model Tree

Base

poolside/Laguna-S-2.1

Quantized

this model

Input Modalities

Text

Output Modalities