brandonmusic

GLM-5.2-NVFP4-REAP-Recall-N172

Why this exists

Standard REAP compresses GLM-5.2 well for code, agentic, and reasoning workloads — but closed-book factual recall collapses: the published narrow-calibrated REAPs answer Lexington for the capital of Kentucky and loop on Marbury v. Madison. The cause is not the model and not the serving stack — it's a calibration corpus that excludes knowledge. This checkpoint re-runs the REAP saliency on a knowledge- and legal-inclusive, axis-balanced calibration.

Table with columns: Prompt, Narrow-REAP baseline, This checkpoint (N=172)
Prompt	Narrow-REAP baseline	This checkpoint (N=172)
What is the capital of Kentucky?	Lexington	Frankfort
In one sentence, what did Marbury v. Madison establish?	(empty / repetition loop)	judicial review — the Supreme Court's authority to declare laws unconstitutional
What is the capital of Texas?	—	Austin

All reasoning prompts still pass (8/8 traps; pipes/syllogism/discount/anticipatory-repudiation all finish=stop).

What I did specifically

Diagnosed it as calibration, not the model — A/B'd kernels, MTP, DCP, image, sampling. Same prompts, same parser, same image: behavior tracked the calibration corpus, not anything else.
Built a 4-axis balanced calibration — 12,228 samples, 3,057 per axis, max 16,384 tokens, no truncation, no packing:
- Axis 1 — General knowledge: C4, Wikipedia, MMLU-aux, TriviaQA, Natural Questions
- Axis 2 — Legal: 1,528 CAP markdown cases + a live Neo4j legal KG (300 headnotes, 390 statutes, 373 case summaries, 113 fact-atoms, 353 worked-examples). fallback_used: false.
- Axis 3 — Code/agentic: evol-codealpaca, Magicoder, xLAM, SWE-smith
- Axis 4 — Reasoning/termination: terminating <think>…</think> traces, </think>-region weighted ×6
Built a real block-wise NVFP4 saliency runner — the published REAP loader can't consume glm_moe_dsa + modelopt NVFP4, and a whole-model vLLM load OOMs the intact 435 GB before any forward. The runner chunks decoder layers into VRAM, dequants NVFP4 → BF16 in place, runs the GlmMoeDsaNaiveMoe modules (explicit per-expert outputs — the ideal saliency hook), captures , frees the chunk.

Quick start

bash

# Pull serving imagedocker pull verdictai/glm52-nvfp4-dcpmtp:v3.3 # Download model (294 GB)huggingface-cli download brandonmusic/GLM-5.2-NVFP4-REAP-Recall-N172 \  --local-dir $HOME/models/GLM-5.2-NVFP4-REAP-Recall-N172 # Serve on 4x RTX PRO 6000 96GB (sm120)docker run -d --name glm52-reap-recall \  --gpus all --runtime nvidia --ipc host --shm-size 32g --network host \  --ulimit memlock=-1 --ulimit stack=67108864 \  -v "$HOME/models":/models-archive:ro -v "$HOME/.cache/glm52-b12x":/cache \  -e CUDA_VISIBLE_DEVICES=0,1,2,3 -e CUDA_DEVICE_ORDER=PCI_BUS_ID -e CUTE_DSL_ARCH=sm_120a \  -e HF_HUB_OFFLINE=1 -e NCCL_IB_DISABLE=1 -e NCCL_P2P_LEVEL=SYS -e NCCL_PROTO=LL,LL128,Simple \  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \  -e VLLM_USE_AOT_COMPILE=1 -e VLLM_USE_BREAKABLE_CUDAGRAPH=0 -e VLLM_USE_FLASHINFER_SAMPLER=1 \  -e B12X_MHC_MAX_TOKENS=16384 -e VLLM_USE_B12X_WO_PROJECTION=1 -e VLLM_USE_B12X_MHC=1 \  -e VLLM_USE_B12X_FP8_GEMM=1 -e VLLM_USE_B12X_MOE=1 -e VLLM_USE_B12X_SPARSE_INDEXER=1 \  -e VLLM_USE_V2_MODEL_RUNNER=1 -e VLLM_USE_FUSED_MOE_GROUPED_TOPK=1 \  -e VLLM_PCIE_ALLREDUCE_BACKEND=b12x -e VLLM_ENABLE_PCIE_ALLREDUCE=1 \  -e B12X_MLA_SM120_UNIFIED=1 -e USES_B12X=True -e B12X_DENSE_SPLITK_TURBO=1 -e B12X_W4A16_TC_DECODE=1 \  -e B12X_MOE_FORCE_A16=1 -e VLLM_DCP_GLOBAL_TOPK=1 -e VLLM_DCP_SHARD_DRAFT=1 \  verdictai/glm52-nvfp4-dcpmtp:v3.3 \  python -m vllm.entrypoints.cli.main serve /models-archive/GLM-5.2-NVFP4-REAP-Recall-N172 \    --served-model-name glm-5.2-nvfp4 --host 0.0.0.0 --port 9405 \    --kv-cache-dtype fp8 --block-size 256 --load-format safetensors \    --tensor-parallel-size 4 --decode-context-parallel-size 4 --moe-backend b12x --linear-backend auto \    --gpu-memory-utilization 0.92 --max-model-len 200000 --max-num-seqs 16 \    --enable-chunked-prefill --enable-prefix-caching --max-num-batched-tokens 8192 \    --max-cudagraph-capture-size 64 --attention-backend B12X_MLA_SPARSE \    --compilation-config '{"custom_ops":["all"],"cudagraph_mode":"PIECEWISE"}' \    --enable-flashinfer-autotune \    --hf-overrides '{"use_index_cache":true,"index_topk_pattern":"FFFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSS"}' \    --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \    --speculative-config '{"method":"mtp","num_speculative_tokens":5,"draft_sample_method":"probabilistic","moe_backend":"b12x","use_local_argmax_reduction":true}' # Sanity checkcurl -s http://127.0.0.1:9405/v1/chat/completions \  -H 'Content-Type: application/json' \  -d '{"model":"glm-5.2-nvfp4","messages":[{"role":"user","content":"what is the capital of kentucky?"}]}'

Optional — thinking_token_budget (hard-caps the reasoning loop): mount the four V2 patch files from the repo into the container and add --reasoning-config '{"reasoning_start_str":"<think>","reasoning_end_str":"</think>"}', then pass "thinking_token_budget": N in the request body. Without those patches, leave --reasoning-config off — on a plain image it clobbers glm45's <think> priming and the chat path stops thinking.

Verified serving config (4× RTX PRO 6000 96GB, TP4)

The v3.3 serving config (image verdictai/glm52-nvfp4-dcpmtp:v3.3):

Table with columns: Knob, Value, Notes
Knob	Value	Notes
Tensor parallel	4	4× 96 GB
Decode context parallel	4	the >300K KV path on TP4
GPU memory util	0.92	DCP4 + MTP headroom (GPU1 also carries ~6.7 GB display)
Max model len	200,000	MTP + DCP4 on fp8; raise with no-MTP
Max num batched tokens	8192	vLLM warns 2048 is suboptimal; 8192 fits the KV pool
KV cache dtype

Benchmarks (measured on 4× RTX PRO 6000 96GB)

Note: the numbers below were measured on the prior nvfp4_ds_mla (4-bit MLA KV) config. A v3.3 (fp8) re-bench is pending.

Benchmarked with local-inference-lab/llm-inference-bench, concurrency × context grid. 15/15 cells completed.

Decode throughput (tok/s)

Table with columns: ctx, C=1, C=2, C=4, agg @ C=4
ctx	C=1	C=2	C=4	agg @ C=4
0	80.7	45.4	40.4	161.8
16K	65.6	46.3	38.2	152.7
32K	52.1	43.2	34.2	136.7
64K

Real-prompt single-user (Marbury essay × 5, thinking OFF)

Table with columns: run, tokens, TTFT, decode tok/s
run	tokens	TTFT	decode tok/s
1	1500	0.13s	58.94
2	1500	1.06s	59.93
3	1500	0.14s	57.52
4	1500	0.15s	57.17
5

Prefill (full ingest)

Table with columns: ctx, tokens, TTFT, tok/s
ctx	tokens	TTFT	tok/s
8K	8,199	4.66s	1,761
16K	16,228	10.87s	1,493
32K	32,321	20.43s	1,582
64K	64,513	42.27s	1,526
128K

KV pool: 542,857 tokens · * VRAM peak: 97.93%

Files

87 safetensors shards (294 GB), config.json, model.safetensors.index.json, tokenizer files, generation_config.json, chat_template.jinja
reap_recall_keep_map_with_scores.json — per-expert real saliency scores and the kept-expert map (the replication artifact)
REAP_RECALL_VERDICT.md — full corpus / saliency / validation ledger
serve_glm52_reap_recall.sh — verified launch script

Sampling

python

{  "temperature": 1.0,  "top_p": 0.95,  "repetition_penalty": 1.05,  "stop_token_ids": [154820, 154827, 154829],  "chat_template_kwargs": {"enable_thinking": true, "reasoning_effort": "high"}}

For non-thinking generation, set enable_thinking: false. For Marbury-style essays, max_tokens >= 1500 recommended (thinking-high needs room).

Credits and lineage

Attribution to the sources:

Z.ai — GLM-5.2 base.
Luke Alonso — GLM-5.2-NVFP4 parent checkpoint and B12X kernels used by the documented serving profile.
NVIDIA TensorRT Model Optimizer — NVFP4 format/tooling consumed by the parent and the block-wise saliency adaptation.
Cerebras Research — REAP method and code, paper, and .

bibtex

@inproceedings{lasby2026reap,  title={{REAP} the Experts: Why Pruning Prevails for One-Shot MoE compression},  author={Mike Lasby and Ivan Lazarevich and Nish Sinnadurai and Sean Lie and Yani Ioannou and Vithursan Thangarasa},  booktitle={The Fourteenth International Conference on Learning Representations},  year={2026},  url={https://openreview.net/forum?id=ukGxWd2aDG}}

License

MIT. See the upstream model and source-project licenses for their respective terms.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

brandonmusic

Model Tree

Base

lukealonso/GLM-5.2-NVFP4

Quantized

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

Why this exists

Table with columns: Prompt, Narrow-REAP baseline, This checkpoint (N=172)
Prompt	Narrow-REAP baseline	This checkpoint (N=172)
What is the capital of Kentucky?	Lexington	Frankfort
In one sentence, what did Marbury v. Madison establish?	(empty / repetition loop)	judicial review — the Supreme Court's authority to declare laws unconstitutional
What is the capital of Texas?	—	Austin

All reasoning prompts still pass (8/8 traps; pipes/syllogism/discount/anticipatory-repudiation all finish=stop).

What I did specifically

Diagnosed it as calibration, not the model — A/B'd kernels, MTP, DCP, image, sampling. Same prompts, same parser, same image: behavior tracked the calibration corpus, not anything else.
Built a 4-axis balanced calibration — 12,228 samples, 3,057 per axis, max 16,384 tokens, no truncation, no packing:
- Axis 1 — General knowledge: C4, Wikipedia, MMLU-aux, TriviaQA, Natural Questions
- Axis 2 — Legal: 1,528 CAP markdown cases + a live Neo4j legal KG (300 headnotes, 390 statutes, 373 case summaries, 113 fact-atoms, 353 worked-examples). fallback_used: false.
- Axis 3 — Code/agentic: evol-codealpaca, Magicoder, xLAM, SWE-smith
- Axis 4 — Reasoning/termination: terminating <think>…</think> traces, </think>-region weighted ×6
Built a real block-wise NVFP4 saliency runner — the published REAP loader can't consume glm_moe_dsa + modelopt NVFP4, and a whole-model vLLM load OOMs the intact 435 GB before any forward. The runner chunks decoder layers into VRAM, dequants NVFP4 → BF16 in place, runs the GlmMoeDsaNaiveMoe modules (explicit per-expert outputs — the ideal saliency hook), captures , frees the chunk.

Quick start

bash

# Pull serving imagedocker pull verdictai/glm52-nvfp4-dcpmtp:v3.3 # Download model (294 GB)huggingface-cli download brandonmusic/GLM-5.2-NVFP4-REAP-Recall-N172 \  --local-dir $HOME/models/GLM-5.2-NVFP4-REAP-Recall-N172 # Serve on 4x RTX PRO 6000 96GB (sm120)docker run -d --name glm52-reap-recall \  --gpus all --runtime nvidia --ipc host --shm-size 32g --network host \  --ulimit memlock=-1 --ulimit stack=67108864 \  -v "$HOME/models":/models-archive:ro -v "$HOME/.cache/glm52-b12x":/cache \  -e CUDA_VISIBLE_DEVICES=0,1,2,3 -e CUDA_DEVICE_ORDER=PCI_BUS_ID -e CUTE_DSL_ARCH=sm_120a \  -e HF_HUB_OFFLINE=1 -e NCCL_IB_DISABLE=1 -e NCCL_P2P_LEVEL=SYS -e NCCL_PROTO=LL,LL128,Simple \  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \  -e VLLM_USE_AOT_COMPILE=1 -e VLLM_USE_BREAKABLE_CUDAGRAPH=0 -e VLLM_USE_FLASHINFER_SAMPLER=1 \  -e B12X_MHC_MAX_TOKENS=16384 -e VLLM_USE_B12X_WO_PROJECTION=1 -e VLLM_USE_B12X_MHC=1 \  -e VLLM_USE_B12X_FP8_GEMM=1 -e VLLM_USE_B12X_MOE=1 -e VLLM_USE_B12X_SPARSE_INDEXER=1 \  -e VLLM_USE_V2_MODEL_RUNNER=1 -e VLLM_USE_FUSED_MOE_GROUPED_TOPK=1 \  -e VLLM_PCIE_ALLREDUCE_BACKEND=b12x -e VLLM_ENABLE_PCIE_ALLREDUCE=1 \  -e B12X_MLA_SM120_UNIFIED=1 -e USES_B12X=True -e B12X_DENSE_SPLITK_TURBO=1 -e B12X_W4A16_TC_DECODE=1 \  -e B12X_MOE_FORCE_A16=1 -e VLLM_DCP_GLOBAL_TOPK=1 -e VLLM_DCP_SHARD_DRAFT=1 \  verdictai/glm52-nvfp4-dcpmtp:v3.3 \  python -m vllm.entrypoints.cli.main serve /models-archive/GLM-5.2-NVFP4-REAP-Recall-N172 \    --served-model-name glm-5.2-nvfp4 --host 0.0.0.0 --port 9405 \    --kv-cache-dtype fp8 --block-size 256 --load-format safetensors \    --tensor-parallel-size 4 --decode-context-parallel-size 4 --moe-backend b12x --linear-backend auto \    --gpu-memory-utilization 0.92 --max-model-len 200000 --max-num-seqs 16 \    --enable-chunked-prefill --enable-prefix-caching --max-num-batched-tokens 8192 \    --max-cudagraph-capture-size 64 --attention-backend B12X_MLA_SPARSE \    --compilation-config '{"custom_ops":["all"],"cudagraph_mode":"PIECEWISE"}' \    --enable-flashinfer-autotune \    --hf-overrides '{"use_index_cache":true,"index_topk_pattern":"FFFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSS"}' \    --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \    --speculative-config '{"method":"mtp","num_speculative_tokens":5,"draft_sample_method":"probabilistic","moe_backend":"b12x","use_local_argmax_reduction":true}' # Sanity checkcurl -s http://127.0.0.1:9405/v1/chat/completions \  -H 'Content-Type: application/json' \  -d '{"model":"glm-5.2-nvfp4","messages":[{"role":"user","content":"what is the capital of kentucky?"}]}'

Optional — thinking_token_budget (hard-caps the reasoning loop): mount the four V2 patch files from the repo into the container and add --reasoning-config '{"reasoning_start_str":"<think>","reasoning_end_str":"</think>"}', then pass "thinking_token_budget": N in the request body. Without those patches, leave --reasoning-config off — on a plain image it clobbers glm45's <think> priming and the chat path stops thinking.

Verified serving config (4× RTX PRO 6000 96GB, TP4)

The v3.3 serving config (image verdictai/glm52-nvfp4-dcpmtp:v3.3):

Table with columns: Knob, Value, Notes
Knob	Value	Notes
Tensor parallel	4	4× 96 GB
Decode context parallel	4	the >300K KV path on TP4
GPU memory util	0.92	DCP4 + MTP headroom (GPU1 also carries ~6.7 GB display)
Max model len	200,000	MTP + DCP4 on fp8; raise with no-MTP
Max num batched tokens	8192	vLLM warns 2048 is suboptimal; 8192 fits the KV pool
KV cache dtype

Benchmarks (measured on 4× RTX PRO 6000 96GB)

Note: the numbers below were measured on the prior nvfp4_ds_mla (4-bit MLA KV) config. A v3.3 (fp8) re-bench is pending.

Benchmarked with local-inference-lab/llm-inference-bench, concurrency × context grid. 15/15 cells completed.

Decode throughput (tok/s)

Table with columns: ctx, C=1, C=2, C=4, agg @ C=4
ctx	C=1	C=2	C=4	agg @ C=4
0	80.7	45.4	40.4	161.8
16K	65.6	46.3	38.2	152.7
32K	52.1	43.2	34.2	136.7
64K

Real-prompt single-user (Marbury essay × 5, thinking OFF)

Table with columns: run, tokens, TTFT, decode tok/s
run	tokens	TTFT	decode tok/s
1	1500	0.13s	58.94
2	1500	1.06s	59.93
3	1500	0.14s	57.52
4	1500	0.15s	57.17
5

Prefill (full ingest)

Table with columns: ctx, tokens, TTFT, tok/s
ctx	tokens	TTFT	tok/s
8K	8,199	4.66s	1,761
16K	16,228	10.87s	1,493
32K	32,321	20.43s	1,582
64K	64,513	42.27s	1,526
128K

KV pool: 542,857 tokens · * VRAM peak: 97.93%

Files

87 safetensors shards (294 GB), config.json, model.safetensors.index.json, tokenizer files, generation_config.json, chat_template.jinja
reap_recall_keep_map_with_scores.json — per-expert real saliency scores and the kept-expert map (the replication artifact)
REAP_RECALL_VERDICT.md — full corpus / saliency / validation ledger
serve_glm52_reap_recall.sh — verified launch script

Sampling

python

{  "temperature": 1.0,  "top_p": 0.95,  "repetition_penalty": 1.05,  "stop_token_ids": [154820, 154827, 154829],  "chat_template_kwargs": {"enable_thinking": true, "reasoning_effort": "high"}}

For non-thinking generation, set enable_thinking: false. For Marbury-style essays, max_tokens >= 1500 recommended (thinking-high needs room).

Credits and lineage

Attribution to the sources:

Z.ai — GLM-5.2 base.
Luke Alonso — GLM-5.2-NVFP4 parent checkpoint and B12X kernels used by the documented serving profile.
NVIDIA TensorRT Model Optimizer — NVFP4 format/tooling consumed by the parent and the block-wise saliency adaptation.
Cerebras Research — REAP method and code, paper, and .

bibtex

@inproceedings{lasby2026reap,  title={{REAP} the Experts: Why Pruning Prevails for One-Shot MoE compression},  author={Mike Lasby and Ivan Lazarevich and Nish Sinnadurai and Sean Lie and Yani Ioannou and Vithursan Thangarasa},  booktitle={The Fourteenth International Conference on Learning Representations},  year={2026},  url={https://openreview.net/forum?id=ukGxWd2aDG}}

License

MIT. See the upstream model and source-project licenses for their respective terms.

GLM-5.2-NVFP4-REAP-Recall-N172

README

Why this exists

What I did specifically

Quick start

Verified serving config (4× RTX PRO 6000 96GB, TP4)

Benchmarks (measured on 4× RTX PRO 6000 96GB)

Decode throughput (tok/s)

Real-prompt single-user (Marbury essay × 5, thinking OFF)

Prefill (full ingest)

Files

Sampling

Credits and lineage

License

Explore FriendliAI today

README

Why this exists

What I did specifically

Quick start

Verified serving config (4× RTX PRO 6000 96GB, TP4)

Benchmarks (measured on 4× RTX PRO 6000 96GB)

Decode throughput (tok/s)

Real-prompt single-user (Marbury essay × 5, thinking OFF)

Prefill (full ingest)

Files

Sampling

Credits and lineage

License