Relativ3pa1n

Laguna-S-2.1-W8A16

README

License: openmdw-1.1

TL;DR

Table

Size	120 GB (from 219 GB BF16)
Format	compressed-tensors W8A16, `group_size=32`, symmetric, Marlin kernels
Serves on	8× RTX 3090 (192 GB VRAM), TP=8
Throughput	~38 tok/s single-stream (CUDA graphs on)
Context / concurrency	256K, 3.18× concurrent
Output	Clean — including technical/markdown content, thinking on
Terminal-Bench (core 0.1.1, 18-task subset, terminus, pass@1)	5/18 = 27.8% (vs W4A16's 4/18 = 22.2%)

Why this quant: the high-fidelity variant

The W4A16 is the small/fast option (69 GB, ~92 tok/s). This W8A16 is the higher-fidelity option (8-bit weights preserve more capability — it scored +1 task on Terminal-Bench). Both serve clean on consumer Ampere (3090).

The non-obvious part: group_size=32 is the Laguna-MoE sweet spot for both 4-bit AND 8-bit. The default g128 produces token-artifacts (byfusion, import torchdec) even at 8-bit; g32 clears them — same fix as the 4-bit. And asymmetric 8-bit is kernel-blocked in vLLM (the Marlin WNA16 MoE backend rejects it — is_supported_config → _supports_quant_scheme returns False for asym-8-bit; llm-compressor #2628), so symmetric g32 is the path. The full quant story (OOM wall → per-layer loop → stale-base → group_size → cudagraph) is in the W4A16 blog.

Serve (vLLM)

vLLM 0.25.1+, 8× 24 GB Ampere:

bash
python -m vllm.entrypoints.openai.api_server \
  --model . --served-model-name laguna \
  --tensor-parallel-size 8 --device-ids 0,1,2,3,4,5,6,7 \
  --quantization compressed-tensors \
  --tool-call-parser poolside_v1 --reasoning-parser poolside_v1 \
  --enable-auto-tool-choice \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --max-model-len 262144 --gpu-memory-utilization 0.90 \
  --max-num-seqs 16 --trust-remote-code \
  --host 0.0.0.0 --port 8085

Key: --quantization compressed-tensors (required), no --enforce-eager (cudagraphs), --gpu-memory-utilization 0.90, poolside_v1 parsers, enable_thinking.

Terminal-Bench — W8A16 vs W4A16 (same 18 tasks)

Table with columns: W4A16 (4-bit), W8A16 (8-bit)
	W4A16 (4-bit)	W8A16 (8-bit)
Score	4/18 = 22.2%	5/18 = 27.8%
Solved	crack-7z-hash.easy, sqlite-with-gcov, prove-plus-comm, sanitize-git-repo	+ path-tracing (W4 failed 4/4; W8 solved all 4 sub-tests)
Near-solve	swe-bench-astropy-2 (8/9)	swe-bench-astropy-2 (10/11) — even closer

W8A16 per-task breakdown:

Table with columns: Task, Result, Reason
Task	Result	Reason
crack-7z-hash.easy	✅ PASS	solved (2 sub-tests)
sqlite-with-gcov	✅ PASS	solved (3)
prove-plus-comm	✅ PASS	solved (4)
sanitize-git-repo	✅ PASS	solved (3)
path-tracing	✅ PASS	solved (4) — W4 failed this
swe-bench-astropy-2	❌ FAIL

License & attribution

Model weights: OpenMDW-1.1 (poolside). Derivative quant — same license.
Built on an 8×3090 rig (192 GB VRAM) with llmcompressor 0.12 / transformers 5.10.1 / vLLM 0.25.1.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Relativ3pa1n

Model Tree

Base

poolside/Laguna-S-2.1

Quantized

this model

Input Modalities

Text

Output Modalities