jackzampolin

Laguna-S-2.1-NVFP4-W4A16-Quality

README

License: openmdw-1.1

Precision placement

Table with columns: Module, Precision
Module	Precision
Dense layer 0	BF16
Routed experts, layers 1-5	BF16
Routed experts, layers 6-8	W8A8 FP8
Routed experts, layers 9-47	static W4A16 NVFP4, group size 16
Attention, shared experts, routers, embeddings, LM head	BF16

39 routed-expert layers use W4A16 NVFP4.
3 routed-expert layers use FP8.
5 routed-expert layers remain BF16.
Effective routed-expert weight precision: 5.9468 bits.
Audited export: 92,469,054,276 bytes, 10 shards, 101,377 tensors.
ModelOpt producer: 0.46.0.dev184+g8813b7001.

Placement was selected with KL-guided AutoQuantize over 512 samples from nemotron-post-training-v3, 512 tokens per sample. The chosen placement was then rebuilt directly from BF16 using FP8 scale sweep and static MSE-scaled W4A16 NVFP4. The exact replay recipe is included as quantization_recipe.yaml.

Five-run KLD

KL direction is KL(BF16 reference || candidate). Every row uses the same hash-pinned 2,048-token prompt, 2,047 next-token positions, and 100,352 logits per position. Runs used TP4, BF16 compute, eager execution, seed 0, no prefix cache, and a fresh engine per run. Quant candidates used the same B12X A16 expert path. Lower is better.

Table with columns: Weights, KV cache, Mean KLD, Sample SD, Range
Weights	KV cache	Mean KLD	Sample SD	Range
This checkpoint	BF16	0.237254	0.004783	0.230693-0.242978
Poolside NVFP4 `216d1f13`	BF16	0.338808	0.009480	0.325988-0.349905
BF16 `a50e85e7`	dynamic FP8 per-token/head

Against the pinned Poolside NVFP4 checkpoint, this export reduced mean KLD by 29.97% with BF16 KV and 20.15% with matched dynamic FP8 KV. KLD values are specific to this model and protocol and should not be compared directly to numbers from other model families.

Reference provenance:

BF16 weights: poolside/Laguna-S-2.1@a50e85e7e0aae7b0a504d156bd36a616ec9fea38
Poolside NVFP4 comparison: poolside/Laguna-S-2.1-NVFP4@216d1f13878dd4e715bc7412848d0f330e95bba6
Prompt token IDs SHA-256: 1f4a417c5159d1ac1202e38c0cb4573955d8ceb2037d8e316b7c3344267959d6
BF16 reference logits SHA-256: beaf167ad42de75af7a0f33554f37c2672092eed175ccaa76cb726e684bca109
Quant config SHA-256: 2102d1290212d218211efbaf0e6b670666a7ea47a11bf255c4980e021899f6af
Safetensors index SHA-256: 0a12b7fa0676018cefe4f4c518f60ee6550d94388273bc7e4cfef027604ae68a

The five aggregate JSON files, placement audit, and checkpoint audit are shipped in this repository.

Serving

This checkpoint is intended for Blackwell GPUs and a W4A16-capable vLLM build. Start with BF16 KV when evaluating quality, then test FP8 KV separately for capacity.

bash
vllm serve jackzampolin/Laguna-S-2.1-NVFP4-W4A16-Quality \
  --trust-remote-code \
  --dtype bfloat16 \
  --quantization auto \
  --tensor-parallel-size 2 \
  --kv-cache-dtype bfloat16

The repository includes Poolside's updated tokenizer from BF16 revision 179ee67cf0fff5391c67fe1a392ea849fa6d643f: thinking is enabled by default, reasoning is preserved across turns, </assistant> is special, and the old 32K generation cap is removed.

Status and limitations

Terminal-Bench 2.1 is in progress using PoolCLI 1.0.13, four attempts per task, and six-hour agent timeouts.
Estonia and LAVD smoke characterization indicated that this release family is sensitive to reasoning behavior; those observations motivated the matched PoolCLI benchmark rather than ad-hoc prompt scoring.
This is an early public release. Please report loops, tool-call failures, and runtime details with the exact KV-cache dtype and serving commit.
Use is governed by the upstream OpenMDW 1.1 license.

Quantized by Jack Zampolin with the local-inference community's quality-first serving workflow.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

jackzampolin

Model Tree

Base

poolside/Laguna-S-2.1

Quantized

this model

Input Modalities

Text

Output Modalities