GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
Relativ3pa1n
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Relativ3pa1n
Model Tree
poolside/Laguna-S-2.1
Input Modalities
Output Modalities
Supported Functionality
| Size | 69 GB (from 219 GB BF16) |
| Format | compressed-tensors W4A16, group_size=32, Marlin kernels |
| Serves on | 8× RTX 3090 (192 GB VRAM), TP=8 |
| Throughput | ~92 tok/s single-stream (CUDA graphs on) |
| Context / concurrency | 256K, 7.33× concurrent |
| Output | Clean — including technical/markdown content, thinking on |
| Terminal-Bench (core 0.1.1, 18-task subset, terminus agent, pass@1) | 4/18 = 22.2% |
Laguna S 2.1 is a top open agentic-coding MoE that fits consumer hardware — but only if you can quantize it without OOMing, pick the right group_size, and serve it without leaving throughput on the floor. The full story (the llmcompressor OOM wall, the per-layer fix, the stale-base red herring, the group_size correction, the cudagraph win) is in LAGUNA-W4A16-BLOG.md.
The short version of the two non-obvious wins:
group_size=32, not 128. The Qwen3.5-style default group_size=128 is too coarse for Laguna's 256-expert MoE — large per-group error flips logits at token boundaries and splices stray number/special tokens into technical + markdown output. Poolside's own reference INT4 uses g32. At g32 the corruption disappears. (Set via QuantizationModifier's config_groups, not scheme — the scheme field only accepts preset names.)--enforce-eager removed). With ~8B active params, per-step Python + MoE-routing-dispatch overhead dominates the tiny compute in eager mode. Cudagraphs fuse it out: ~7 tok/s → ~92 tok/s (12.5×) at the cost of a hair less KV (7.65× → 7.33× concurrency @ 256K).vLLM 0.25.1+, 8× 24 GB Ampere. The cudagraph config is the whole game:
bash
python -m vllm.entrypoints.openai.api_server \--model . --served-model-name laguna \--tensor-parallel-size 8 --device-ids 0,1,2,3,4,5,6,7 \--quantization compressed-tensors \--tool-call-parser poolside_v1 --reasoning-parser poolside_v1 \--enable-auto-tool-choice \--default-chat-template-kwargs '{"enable_thinking": true}' \--max-model-len 262144 --gpu-memory-utilization 0.90 \--max-num-seqs 16 --trust-remote-code \--host 0.0.0.0 --port 8085
Key flags: --quantization compressed-tensors (required), no --enforce-eager (cudagraph = the 12× lever), --gpu-memory-utilization 0.90 (higher over-reserves KV and OOMs sampler warmup), poolside_v1 tool/reasoning parsers, enable_thinking. Full runnable script: Laguna-S-2.1-W4A16-vLLM.sh.
18-task random subset of terminal-bench-core==0.1.1, terminus agent, pass@1, default turn budget, 900s agent cap. Honest sample — not the full suite.
| Task | Result | Failed sub-tests | Reason |
|---|---|---|---|
| crack-7z-hash.easy | ✅ PASS | — | solved (2 sub-tests) |
| sqlite-with-gcov | ✅ PASS | — | solved (3 sub-tests) |
| prove-plus-comm | ✅ PASS | — | solved (4 sub-tests) |
| sanitize-git-repo | ✅ PASS | — | solved (3 sub-tests) |
Notable: several fails were near-solves (e.g. one task passed 8/9 sub-tests), and the hard infrastructure tasks (QEMU / kernel-build / maze / video-extraction) hit the agent timeout — expected for an 8B-active model on a frontier-calibrated benchmark (the TB paper has frontier models <65% on the full suite).
Laguna drives the terminus agent with structured per-episode reasoning. On the hf-model-inference task (build a Flask sentiment API around a HF model), episode 1:
state_analysis: "Packages installed successfully (torch, transformers, flask). Now need to download the model to
/app/model_cache/sentiment_modeland create the Flask API." explanation: "Will download the Hugging Face model using the transformers library and save it to the specified cache directory. Then create the Flask API application." commands:mkdir -p /app/model_cache/sentiment_model→python3 -c "from transformers import AutoTokenizer, AutoModelForSequenceClassification; … model.save_pretrained('/app/model_cache/sentiment_model')"
Concrete evidence the quant retains agentic reasoning, not just chat fluency.
| file | what |
|---|---|
model-*.safetensors (4 shards) | the W4A16-g32 weights (69 GB) |
config.json | includes quantization_config (group_size=32, Marlin) |
Laguna-S-2.1-W4A16-vLLM.sh | the vLLM serve recipe (cudagraph config) |
LAGUNA-W4A16-BLOG.md | full quant writeup (5 walls) |
w4a16-quant-laguna-s-2.1-perlayer.py | the per-layer quantizer (restricted-RAM) |
| swe-bench-astropy-2 | ❌ FAIL | test_roundtrip[True] | near-solve — 8/9 sub-tests passed |
| security-vulhub-minio | ❌ FAIL | test_command_output_content | sub-test failed (1/1) |
| download-youtube | ❌ FAIL | test_output_file_exists, test_correct_video | sub-tests failed (2/2) |
| path-tracing | ❌ FAIL | image compile/run/similarity (4) | sub-tests failed (4/4) |
| blind-maze-explorer-5x5 | ❌ FAIL | test_maze_map_contents | agent 900s timeout |
| qemu-alpine-ssh | ❌ FAIL | test_sshpass | agent 900s timeout (hard infra) |
| build-linux-kernel-qemu | ❌ FAIL | TypeError in test | agent 900s timeout (hard infra) |
| sqlite-db-truncate | ❌ FAIL | test_json_data | agent 900s timeout |
| polyglot-c-py | ❌ FAIL | — | agent 900s timeout |
| extract-moves-from-video | ❌ FAIL | — | agent 900s timeout (video processing) |
| chess-best-move | ❌ FAIL | test_move_correct | agent 900s timeout |
| password-recovery | ❌ FAIL | test_recovery_file_exists, test_password_match | agent 900s timeout |
| pytorch-model-cli.hard | ❌ FAIL | — | test-harness 120s timeout |
| cron-broken-network | ❌ FAIL | — | test output unparseable (tests didn't run) |