Evaluation
All benchmarks were served via SGLang and scored with lm-evaluation-harness on the same hardware and
harness for both NVFP4 and BF16 (generative / chain-of-thought where applicable; max_gen_toks raised
to fit the reasoning chains — lm-eval's default 256 truncates them and tanks the scores).
Table with columns: Benchmark, GLM-5.2-NVFP4 (410 GB), GLM-5.2 BF16 (1507 GB), Δ| Benchmark | GLM-5.2-NVFP4 (410 GB) | GLM-5.2 BF16 (1507 GB) | Δ |
|---|
| GPQA-Diamond (CoT, flexible) | 69.70 | 69.70 | 0.00 |
| MATH-500 (minerva) | 86.80 | 86.60 | +0.20 |
| MMLU-Pro (generative, 50/subject) | 81.14 | 82.43 | −1.29 |
| HumanEval (pass@1, instruct) | 94.51 | 95.73 | −1.22 |
| GSM8K (5-shot, flexible) | 92.72 | 94.92 | −2.20 |
NVFP4 holds up strongly on the hard, non-saturated benchmarks: GPQA-Diamond and MATH-500 are within
noise of BF16, and the average degradation across the suite is ~1 point — for a 3.7× smaller checkpoint.
Quantization recipe
- Format: NVFP4 (FP4 weights + FP8 block scales), block/group size 16,
modelopt producer.
- Quantized:
mlp.experts.* (256 routed experts) and mlp.shared_experts.*.
- Kept in BF16 (excluded): all of
self_attn.* — MLA projections (q/kv) and the DSA indexer —
plus the MoE router (mlp.gate) and lm_head. The indexer and MLA attention must stay BF16:
SGLang's deepseek_v2 MLA path (used for glm_moe_dsa) cannot consume NVFP4 attention weights.
- KV cache: not quantized.
- Calibration: 512 samples × 2048 tokens from cnn_dailymail + nvidia/OpenCodeReasoning +
nvidia/OpenMathReasoning.
Serving (SGLang)
Requires SGLang ≥ v0.5.13.post1 (the version that registers GlmMoeDsaForCausalLM).
docker run --runtime=nvidia --gpus '"device=0,1,2,3"' --ipc=host --shm-size=32g \
-v /path/to/GLM-5.2-NVFP4:/model -p 30000:30000 \
lmsysorg/sglang:v0.5.13.post1-cu130 \
sglang serve --model-path /model --tp 4 \
--quantization modelopt_fp4 --moe-runner-backend flashinfer_cutlass \
--context-length 32768 --mem-fraction-static 0.85 \
--tool-call-parser auto --trust-remote-code --host 0.0.0.0 --port 30000
GPU memory. The weights are ~410 GB, so per-GPU footprint depends on TP:
Table with columns: Tensor parallel, Weights / GPU, Suitable GPUs| Tensor parallel | Weights / GPU | Suitable GPUs |
|---|
--tp 4 | ~110 GB | ≥128 GB cards — H200 (141 GB, tight KV), B200 / B300, MI300X (192 GB) |
--tp 8 | ~55 GB | 80 GB cards — 8× H100 or A100-80GB |
So 80 GB GPUs need --tp 8, not --tp 4 (110 GB of weights can't fit in an 80 GB card). Lower
--mem-fraction-static if KV-cache space is tight. Use a generous max_tokens at inference — GLM-5.2 is
a reasoning model and its <think> chains can be long.
Notes
- Quantized with
nvfp4 + a small build_quant_cfg exclusion that keeps self_attn.* in BF16 (required
for SGLang's MLA path). Same overall pipeline as our MiniMax-M3-NVFP4.
- License inherited from the base model (MIT, Zhipu AI).
