Quality (teacher-forced perplexity, same harness for all models)
Table with columns: eval set, this repo, REAP-504B-v2 (168/256 experts, 308 GB), GLM-5.2-504B-Nvidia (NVFP4 REAP, 318 GB)| eval set | this repo | REAP-504B-v2 (168/256 experts, 308 GB) | GLM-5.2-504B-Nvidia (NVFP4 REAP, 318 GB) |
|---|
| code+docs+medical, 159k tok | 2.5223 | 2.6961 | 2.8784 |
| wikitext-2 slice | 3.7024 | 6.8941 | 9.0968 |
| fresh GitHub (post-2026-06) | 2.2910 | 2.2603 | 2.3460 |
| code+docs, 46.7k tok | 2.7393 | 2.7347 | 2.8661 |
All 256 experts per layer remain routable here (nothing is pruned); the
REAP comparisons drop 88 experts/layer, which shows up as large regressions
on prose and medical/general-knowledge content.
Sibling variants trading context for more NVFP4:
500k / 48% hot ·
250k / 57% hot
🐳 Run with Docker on SM120 (recommended)
The easiest way to serve this model on 4x RTX PRO 6000 (SM120) is the
pre-configured Docker image from
jarrelscy/vllm-glm52-sm120
— a vLLM fork with the hybrid NVFP4+AQLM MoE support and DSpark
speculative decode under pipeline parallelism already committed (no
runtime patching):
git clone -b glm52-sm120 https://github.com/jarrelscy/vllm-glm52-sm120
cd vllm-glm52-sm120
docker build -f Dockerfile.glm52-sm120 -t glm52-sm120 .
# 1M context (PP4, full window):
docker run --gpus all --ipc=host -p 8001:8000 \
-v /path/to/weights:/models/1m:ro \
-e PARALLEL=pp4-1m glm52-sm120
# Fastest decode (~59.6 tok/s): TP4 + DSpark speculator, ~200K context:
docker run --gpus all --ipc=host -p 8001:8000 \
-v /path/to/weights:/models/1m:ro \
-e PARALLEL=tp4-dspark glm52-sm120
# Balanced (~35 tok/s, ~223K): TP2xPP2 + DSpark:
docker run --gpus all --ipc=host -p 8001:8000 \
-v /path/to/weights:/models/1m:ro \
-e PARALLEL=tp2pp2 glm52-sm120
Note: 1M + DSpark is not possible on 4x96GB (the drafter KV cache does
not co-fit); use pp4-1m for the full window.
Serving
Requires a patched vLLM (patch included in code/vllm-patches/, applies to
vllm-project/vllm @ 95a248fae). One-command setup + serve:
hf download jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid code/run_rtx6000.sh --local-dir .
bash code/run_rtx6000.sh 1m
Manual launch (after applying the patch and downloading this repo):
VLLM_PP_LAYER_PARTITION=21,19,19,19 NCCL_MAX_NCHANNELS=4 NCCL_BUFFSIZE=1048576 \
VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=192 \
vllm serve <repo-dir> --pipeline-parallel-size 4 --gpu-memory-utilization 0.95 \
--kv-cache-dtype fp8_ds_mla --max-model-len 1048576 \
--max-num-seqs 2 --max-num-batched-tokens 2048 --enforce-eager
code/ contains the full production pipeline (routing-stats collection,
per-expert assignment solver, checkpoint builders), the vLLM patch, the
calibration routing statistics, and SETUP.md.
Quantized from lukealonso/GLM-5.2-NVFP4.
AQLM codebooks fit with residual k-means + least-squares scale refit
(no PV-tuning yet). Validated on B200 (SM100, FlashMLA-sparse backend);
SM120 uses FLASHINFER_MLA_SPARSE_SM120 — same KV format, smoke-test on
first deploy.
