🐳 Serve on SM120 (recommended)
Use the vLLM fork
jarrelscy/vllm-glm52-sm120
— the hybrid NVFP4+AQLM MoE support, SM120 sparse-MLA (DSA) fixes, DCP, and both
MTP / DSpark speculative decode are already committed (no runtime patching).
The full per-config flag reference (with this variant's --max-model-len baked
in) is in SERVING.md.
Decode throughput is set by the config, not the checkpoint — all variants
decode identically per-config (validated on the 1M sibling: needle-in-haystack
PASS @ 749K). The variants differ only in context ceiling; this checkpoint
tops out near ~500k, so use MAXLEN=500000.
git clone -b glm52-sm120 https://github.com/jarrelscy/vllm-glm52-sm120
cd vllm-glm52-sm120
docker build -f Dockerfile.glm52-sm120 -t glm52-sm120 .
# ★ PREFERRED — ~500k window WITH lossless MTP spec, TP speed + CUDA graphs
# TP4 + DCP4 + native MTP (ns5) + PIECEWISE graphs, ~55 tok/s short:
docker run --gpus all --ipc=host -p 8001:8001 \
-v /path/to/weights:/models/1m:ro \
-e PARALLEL=tp4-1m-mtp -e MAXLEN=500000 glm52-sm120
# Full window, PP4 no spec, ~18.7 tok/s:
docker run --gpus all --ipc=host -p 8001:8001 \
-v /path/to/weights:/models/1m:ro \
-e PARALLEL=pp4-1m -e MAXLEN=500000 glm52-sm120
# Fastest decode (~66/53 tok/s easy/code), TP4 + DSpark, short context:
docker run --gpus all --ipc=host -p 8001:8001 \
-v /path/to/weights:/models/1m:ro -e PARALLEL=tp4-dspark glm52-sm120
Prefill: ~1.2–1.6K tok/s (measured on the PP4/MTP configs). Prefix caching is
on, so a long prompt is ingested once and reused across requests.
tp4-1m-mtp gives ~1M+spec on the 1M sibling; here the same config gives
~500k+spec (DCP — a stock vLLM feature for MLA models — shards the MLA latent KV by sequence; MTP's MLA-shaped draft KV
shares that sharded latent). Override draft depth with -e NUM_SPEC=7
(structured/code) or -e NUM_SPEC=2 (general). The heavier NVFP4 weight per GPU
leaves less KV room than the 1M variant, so if a config OOMs, drop
--max-model-len or nudge --gpu-memory-utilization.
code/ in the main repo
has the full production pipeline (routing stats, assignment solver, checkpoint
builders) and SETUP.md. Quantized from
lukealonso/GLM-5.2-NVFP4.
