Quality (teacher-forced perplexity, same harness for all models)
Table with columns: eval set, this repo, REAP-504B-v2 (168/256 experts, 308 GB), GLM-5.2-504B-Nvidia (NVFP4 REAP, 318 GB)| eval set | this repo | REAP-504B-v2 (168/256 experts, 308 GB) | GLM-5.2-504B-Nvidia (NVFP4 REAP, 318 GB) |
|---|
| code+docs+medical, 159k tok | 2.5223 | 2.6961 | 2.8784 |
| wikitext-2 slice | 3.7024 | 6.8941 | 9.0968 |
| fresh GitHub (post-2026-06) | 2.2910 | 2.2603 | 2.3460 |
| code+docs, 46.7k tok | 2.7393 | 2.7347 | 2.8661 |
All 256 experts per layer remain routable (nothing pruned); the REAP comparisons drop
88 experts/layer → large regressions on prose and medical/general-knowledge content.
Sibling variants trading context for more NVFP4:
500k / 48% hot ·
250k / 57% hot
Serving (4x RTX PRO 6000, SM120 — TP4 + DCP4)
1. The vLLM fork
Not stock vLLM. Use the pre-patched fork (no runtime patching):
github.com/jarrelscy/vllm-glm52-sm120 (branch glm52-sm120)
It builds on a recent upstream vLLM that already ships DCP
(--decode-context-parallel-size, decode context parallelism for MLA models,
vLLM PR #46076) and native MTP speculative decode; the fork adds the hybrid
NVFP4+AQLM MoE loader/kernels, the SM120 sparse-MLA (DSA) fixes that make DCP
and MTP coherent on RTX PRO 6000, and PIECEWISE-graphed draft decodes.
git clone -b glm52-sm120 https://github.com/jarrelscy/vllm-glm52-sm120
cd vllm-glm52-sm120 && docker build -f Dockerfile.glm52-sm120 -t glm52-sm120 .
2. Recommended (and default) config — tp4-1m-mtp
TP4 + DCP4(ag_rs) + native MTP (ns=3, IndexShare) + PIECEWISE CUDA graphs +
bit-exact gemv kernels + forced NCCL P2P + tool calling. Coherent ~1M window
at TP speed with lossless MTP self-speculation. This is the default — the
docker entrypoint applies the whole 2026-07-12 promoted stack automatically:
docker run --gpus all --ipc=host -p 8001:8001 \
-v /path/to/weights:/models/1m:ro \
-e PARALLEL=tp4-1m-mtp glm52-sm120
Equivalent bare launch:
export NCCL_MAX_NCHANNELS=4 NCCL_BUFFSIZE=1048576 VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=192
export NCCL_P2P_LEVEL=SYS VLLM_MTP_INDEX_SHARE=1 GLM_MOE_LANE_ROWS=1 GLM_NVFP4_LUT256=1
vllm serve <repo-dir> \
--tensor-parallel-size 4 \
--decode-context-parallel-size 4 --dcp-comm-backend ag_rs \
--speculative-config '{"method":"deepseek_mtp","num_speculative_tokens":3}' \
--compilation-config '{"mode":3,"cudagraph_mode":"PIECEWISE"}' \
--gpu-memory-utilization 0.97 --kv-cache-dtype fp8_ds_mla \
--max-model-len 950000 --max-num-seqs 2 --max-num-batched-tokens 4096 \
--no-enable-flashinfer-autotune \
--enable-auto-tool-choice --tool-call-parser glm47 \
--served-model-name glm-5.2 --port 8001
cudagraph_mode MUST be PIECEWISE for DCP+spec. Do not add --enforce-eager.- ns=3 is the tuned optimum: single-stream verify batch = ns+1 = 4 hits the exact graph-4 replay; ns≥4 pads verify to graph-8 (2× work). Override with
-e NUM_SPEC=.
- Tool/function calling is on (
glm47 parser); the chat template ships <tool_call> tags. -e ENABLE_TOOLS=0 to disable.
- The 2026-07-12 stack (every piece verified lossless against a 64K-token temp-0 teacher-forced golden + 1M-capability + MTP-acceptance gates, individually and combined):
VLLM_MTP_INDEX_SHARE=1 — MTP draft steps reuse the verify-anchored DSA indexer top-k instead of recomputing per draft forward (the recompute also produced bad draft top-k; fixing it lifts count-workload acceptance 0.43→0.96). Draft-only ⇒ lossless by rejection sampling. Biggest single win, grows with context.
3. Alternatives (set -e PARALLEL=)
Table with columns: config, what, window / speed| config | what | window / speed |
|---|
| tp4-1m-mtp ★ default | TP4+DCP4+MTP ns3 + graphs + tools | ~1M + lossless spec |
| tp4-1m | TP4+DCP4, no spec | ~1M — best multi-stream throughput |
| tp4-dspark | TP4 + DSpark | ~247K — fastest short-context |
- Single-stream decode (
tp4-1m-mtp default stack): short context ~73 tok/s counting / 64 code / 52 prose; at 123K context ~44 / 41 / 32 tok/s; ~44 code at ~490K. All lossless (64K temp-0 golden-gated).
- Fresh prefill: ~1,200–1,270 tok/s at 115–123K (a ~950K prompt ≈ 12.5 min); prefix-cached follow-ups are near-instant.
- Concurrency / throughput @32K (aggregate decode tok/s): MTP saturates ~64; no-spec
tp4-1m scales to ~128 (~2×) — spec is a single-user latency play, no-spec is the multi-user throughput play. Use tp4-1m + raise --max-num-seqs for serving many streams.
Measure spec decode with non-streamed responses — under MTP, vLLM bundles multiple accepted tokens into one SSE delta, so chunk-counting undercounts by ~the acceptance factor.
code/ contains the production pipeline (routing stats, per-expert assignment solver,
checkpoint builders) and SETUP.md. Quantized from
lukealonso/GLM-5.2-NVFP4.
