Why this model
- Half the footprint, FP8 quality. ~405 GB of weights (down from ~1.49 TB BF16) serve one replica on
4×H200 instead of 8 — freeing half the fleet, or two replicas per node — and eval matches the FP8
baseline within noise across reasoning, instruction-following, long-context, and agentic coding.
- Fastest interactive serving among GLM-5.2 quants on Hopper. In a matched benchmark (every model with
MTP on, same box, same vLLM, same harness): +8% vs nvidia NVFP4 and +33% vs zai FP8 at concurrency 1,
with TTFT of 215 ms vs 632/1258 ms.
- Honest trade-off. MTP's draft/verify overhead stops paying off once the batch saturates — at c32 the
NVFP4/FP8 quants are ~11% faster. If your workload is fully-saturated batch serving, pick by that row.
Throughput — matched MTP-on comparison (8×H200, vLLM v0.23.0, same harness)
All three models served identically (--speculative-config '{"method":"mtp","num_speculative_tokens":5}',
TP=8, fp8 KV); benchmarked with vllm bench serve (openai-chat endpoint, random 1024-in/512-out,
num-prompts = 8×concurrency). Measured 2026-07-02 on AWS p5e.48xlarge.
Table with columns: concurrency, This (W4A16 + MTP), nvidia NVFP4 + MTP, zai FP8 + MTP| concurrency | This (W4A16 + MTP) | nvidia NVFP4 + MTP | zai FP8 + MTP |
|---|
| 1 — tok/s (TTFT) | 125.7 (215 ms) | 116.3 (632 ms) | 94.4 (1258 ms) |
| 8 | 495.7 (319 ms) | 455.5 (422 ms) | 394.2 (742 ms) |
| 32 | 828.4 (413 ms) | 925.2 (403 ms) | 921.4 (412 ms) |
MTP spec-decode acceptance was ~28% for all three models on this synthetic workload (higher, ~46–52%, on
natural eval traffic) — the draft head performs the same across quants, so this is a clean quant-vs-quant
comparison. Note NVFP4 is a Blackwell-native format measured here on Hopper, where it has no FP4 tensor
cores; treat its column as a Hopper-deployment number.
Purpose
GLM-5.2 (744B-parameter MoE) in BF16 needs ~1.49 TB of weights — eight 141 GB H200s, fully occupied, to serve
one replica. The goal of this artifact is a smaller-footprint variant that matches FP8 quality so the model
runs on four H200s instead of eight (freeing half the fleet, or two replicas per node), while keeping the
MTP draft head for speculative-decode speedups. It is a deployment-efficiency artifact, not a new model — all
capability comes from the base GLM-5.2.
Details
Table with columns: Field, Value| Field | Value |
|---|
| Base model | zai-org/GLM-5.2 (BF16) |
| Architecture | GlmMoeDsaForCausalLM — 744B MoE, ~40B active, MLA + DeepSeek Sparse Attention, 1M context |
| Weight quantization | W4A16, INT4, asymmetric, group-size 128 (GPTQ, compressed-tensors), routed experts only |
| Kept in BF16 | attention, dense layers (0–2), shared experts, router/gate, embeddings, lm_head, MTP layer 78 |
| MTP | layer 78 preserved at BF16 for spec-decode (num_speculative_tokens=5) |
| Calibration |
The "FP8" sometimes seen in the filename refers to the fp8 KV-cache used at serving time, not the
weights — the weights are INT4 (W4A16) and the MTP layer is BF16.
Evaluation — vs the FP8 baseline (same harness, 8×H200)
Measured against zai-org/GLM-5.2-FP8 under an identical setup (generative tasks via chat-completions with a
16,384-token generation budget for the reasoning CoT; matched serve config with --reasoning-parser).
Quality is measured with speculative decoding off, where it is exact — MTP changes latency, not outputs.
Table with columns: Task, This (W4A16+MTP), FP8 baseline| Task | This (W4A16+MTP) | FP8 baseline |
|---|
| GSM8K (strict) | 0.960 | 0.955 |
| IFEval (prompt-strict / inst-strict) | 0.909 / 0.911 | 0.891 / 0.903 |
| MATH-500 (math-verify) | 0.954 | 0.958 |
| RULER @ 32K | 0.832 | 0.831 |
| RULER @ 64K | 0.841 | 0.813 |
| SWE-bench Verified (mini-SWE-agent + official grading) | (410/500) |
Quantization preserves quality: scores track the FP8 baseline within run-to-run noise on reasoning,
instruction-following, long-context retrieval, and agentic coding. (MMLU-Pro: FP8 full-set = 0.820; the W4A16
subset run was not completed — the verdict was already conclusive from the six tasks above. RULER used 50
samples per sub-task, not the full 500.)
Long context: serves at max_model_len=1,048,576 on 8×H200 and correctly retrieved a needle from a
~936K-token prompt (MLA + DSA compress the KV cache enough to fit 1M in the memory free after weights).
On 4×H200 it serves 128K validated (single-stream engine ceiling ~239K at gpu-memory-utilization=0.92;
256K overflows the post-weights KV budget) and retrieved a 64K needle at both mid- and end-placement.
MTP: speculative-decode acceptance ~46–52% aggregate (~95% at draft position 0) on natural eval traffic on
8×H200, confirming the injected BF16 MTP layer is healthy. On 4×H200 (TP=4, 128K) aggregate acceptance is ~38%
(7,848/20,765 draft tokens, mean accept-length ~2.9) — mildly lower under the tighter memory split but still a
net speedup.
Serving (vLLM ≥ 0.23, Hopper / H200)
The asymmetric W4A16 MoE requires expert parallelism (--enable-expert-parallel); plain tensor-parallel
trips a Marlin scale-sharding bug. The DSA indexer needs an nvcc ≥ 12.8 toolchain (CUDA_HOME). Validated on
vLLM v0.23.0 (newer versions changed the DSA indexer layout — v0.24+ currently fails to load this checkpoint's
per-layer indexer weights; pin v0.23.x until upstream support lands).
8×H200 (up to 1M context):
vllm serve <repo> \
--tensor-parallel-size 8 --enable-expert-parallel \
--kv-cache-dtype fp8 \
--speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
--reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
--max-model-len 1048576 --gpu-memory-utilization 0.90 --trust-remote-code
4×H200 (the footprint win, 128K validated / ~239K single-stream ceiling — 1M needs all 8):
vllm serve <repo> --tensor-parallel-size 4 --enable-expert-parallel \
--kv-cache-dtype fp8 --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
--reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
--max-model-len 32768 --gpu-memory-utilization 0.92 --trust-remote-code
Serving on Blackwell — 8×RTX PRO 6000 (SM120)
Validated end-to-end on 8× RTX PRO 6000 (96 GB, SM 12.0, PCIe): quality matches the H200 deployment
(GSM8K 0.948, IFEval 0.909/0.920, MATH-500 0.954 math-verify, RULER@32K 0.918 / @64K 0.826 — all within
margin of the H200 column above), zero token corruption, with MTP speculative decoding and cudagraphs
both working. Throughput on this config:
Table with columns: concurrency, 1, 4, 8, 16, 32, 64| concurrency | 1 | 4 | 8 | 16 | 32 | 64 |
|---|
| output tok/s | 50 | 148 | 280 | 400 | 613 | 989 |
For reference, nvidia/GLM-5.2-NVFP4 on the same box (MTP off — a matched MTP-on NVFP4 run was not
completed) measures 36 tok/s at c=1 and 447 at c=64. Single-stream speed is memory-bandwidth-bound on this
hardware (~50 vs ~126 tok/s on H200); the sweet spot is concurrent/batch serving.
Why SM120 needs a different recipe: no SM120 sparse-MLA kernel supports GLM-5.2's DSA head layout, so
DSA sparse attention is disabled and the model serves through dense TRITON_MLA. That takes four one-line
patches on the official vllm/vllm-openai:glm52 image:
FROM vllm/vllm-openai:glm52
# 1. Disable DSA sparse attention (no SM120 sparse-MLA backend for this head size)
RUN sed -i 's/self\.is_v32 = hasattr(config, "index_topk")/self.is_v32 = False/g' \
/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v2.py
# 2-4. Skip the now-orphaned DSA indexer weights during load (deepseek_v2.py,
# deepseek_mtp.py, glm4_moe_mtp.py): guard `param = params_dict[name]` with
# `if name not in params_dict: continue`
RUN python3 - <<'EOF'
import re
base = '/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/'
for f in ('deepseek_v2.py', 'deepseek_mtp.py', 'glm4_moe_mtp.py'):
p = base + f
src = open(p).read()
src = re.sub(r'(\s+)param = params_dict\[name\]',
r'\1if name not in params_dict:\n\1 continue\n\1param = params_dict[name]', src)
open(p, 'w').write(src)
EOF
Then serve (docker build -t glm52-mtp-sm120 . first):
docker run -d --gpus all --ipc=host --shm-size 16g \
-v /path/to/GLM-5.2-W4A16-MTP:/model:ro -p 8000:8000 \
-e NCCL_P2P_DISABLE=1 -e VLLM_USE_DEEP_GEMM=0 -e VLLM_MOE_USE_DEEP_GEMM=0 \
glm52-mtp-sm120 /model \
--tensor-parallel-size 8 --enable-expert-parallel \
--attention-backend TRITON_MLA \
--speculative-config '{"method":"mtp","num_speculative_tokens":1}' \
--compilation-config '{"cudagraph_mode":"PIECEWISE","cudagraph_capture_sizes":[2,4,8,16,32,64],"max_cudagraph_capture_size":64}' \
--max-model-len 131072 --max-num-seqs 64 --gpu-memory-utilization 0.92 \
--reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
--disable-custom-all-reduce --trust-remote-code
Every deviation from the Hopper command is load-bearing:
- bf16 KV cache (no
--kv-cache-dtype fp8) — fp8 KV overflows SM120's per-SM shared memory in the
TRITON_MLA kernel (102,400 > 101,376 bytes).
PIECEWISE cudagraph mode — FULL decode graphs + MTP produce degenerate output on TRITON_MLA (its
decode kernel only handles single-token queries; MTP verify sends multi-token queries). PIECEWISE routes
them through the prefill path correctly and still gives ~10× over eager at c=1.- MTP
num_speculative_tokens=1, with cudagraph capture sizes divisible by (1 + k) = 2.
VLLM_USE_DEEP_GEMM=0 — DeepGEMM's attention path doesn't support SM120.--attention-backend TRITON_MLA — the dense-MLA backend that works once DSA is disabled.
A one-shot bootstrap script (HF download → image build → launch, idempotent) exists in the companion
repository as scripts/bootstrap_sm120_glm52_w4a16_mtp.sh.
Method
- GPTQ W4A16 (group-128, asymmetric) on the routed experts only, with attention/dense/MTP/embeddings/
lm_head held at BF16.
calibrate_all_experts=True is required — calibrating only routed experts starves
rarely-activated experts and produces a coherent-looking but degenerate model.
- MTP preservation (Option-Y): GLM-5.2's MTP/nextn layer (index 78) isn't instantiated by
from_pretrained, so quantization never sees it. It is injected back at BF16 from the source checkpoint
after quantization and added to the ignore list so the serving stack treats it as unquantized.
The full recipe, evaluation methodology, and a log of the engineering walls hit and overcome are in the
companion repository (calibration memory limits, MoE sequential-target OOMs, the MTP-loss-on-save issue, the
asymmetric-MoE serving fix, and the Blackwell toolchain gaps).
Limitations
- The interactive edge is +8% vs NVFP4+MTP / +33% vs FP8+MTP at c1; at full saturation (c32) those quants are
~11% faster. Pick by your operating point.
- 1M-context serving requires all 8 H200s; 4×H200 serves up to ~128K (single-stream engine ceiling ~239K),
with MTP acceptance ~38% (vs ~46–52% on 8×H200).
- Asymmetric weights require
--enable-expert-parallel to serve correctly.
- Pin vLLM v0.23.x (v0.24+ DSA-indexer layout change breaks loading).
- On Blackwell SM120 (RTX PRO 6000) use the dedicated recipe above: DSA sparse attention must be disabled
(dense TRITON_MLA), KV cache stays bf16, cudagraphs run in PIECEWISE mode, and MTP is limited to k=1.
Quality is unaffected (validated at parity with H200); single-stream throughput is bandwidth-bound at
~50 tok/s, so size deployments for concurrent traffic. FULL-cudagraph + MTP on TRITON_MLA is an upstream
kernel gap (vllm-project/vllm#21505), not fixable by configuration.
Acknowledgements
Built on zai-org/GLM-5.2 (MIT). Quantized with
llm-compressor; served with vLLM.