GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
aidendle94
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
aidendle94
Model Tree
Base
zai-org/GLM-5.2
Quantized
this model
Input Modalities
Text
Output Modalities
Text
Supported Functionality
Dedicated Endpoints
Container
The checkpoint is 382 GiB and does not fit one GB10 (121 GiB unified memory). It requires four GB10 nodes (TP4) on a fast fabric — 200G RoCE/InfiniBand recommended; Ethernet boots but collectives are slow. Each node needs the driver, Docker, the NVIDIA Container Toolkit, and ~110 GiB free disk.
1. Download the checkpoint to the same path on every node:
bash
hf download aidendle94/GLM-5.2-Hybrid-FP8-MXFP4 --local-dir /models/glm52-hybrid
2. Pull the turnkey image on every node:
bash
docker pull aidendle94/sparkrun-vllm-ds4-gb10:production-hybrid-1.3
vLLM + the b12x GB10 kernels with all required overlays baked in (the hybrid_mxfp4_ct
quant config, index-authoritative loader, NVFP4 KV cache with per-token scaling, sparse-MLA
CKV gather, boot-time kernel prewarm). No host-side patching.
3. Launch one container per node — identical except RANK (0 on head, 1–3 on workers,
--headless on workers). Set HEAD_IP, NIC, and HCA for your fabric:
bash
docker run -d --name glm52-hybrid \--network host --ipc host --shm-size 10gb --gpus all \--cap-add IPC_LOCK --ulimit memlock=-1:-1 \--device /dev/infiniband:/dev/infiniband \-v /models/glm52-hybrid:/hybridmodel:ro \-v $HOME/glm-jit-cache:/cache/jit \-e VLLM_USE_V2_MODEL_RUNNER=1 -e VLLM_USE_B12X_MOE=1 \-e VLLM_USE_B12X_SPARSE_INDEXER=1 \-e VLLM_DCP_SHARD_DRAFT=1 -e VLLM_DCP_GLOBAL_TOPK=1 -e VLLM_DSV4_INDEXER_SP=1 \-e VLLM_B12X_MLA_CKV_GATHER=1 -e VLLM_B12X_MLA_CKV_GATHER_MAX_TOKENS=262144 \-e VLLM_B12X_MLA_DECODE_SPARSE_GATHER=1 -e VLLM_B12X_MLA_DECODE_GATHER_V2=1 \-e VLLM_ADAPTIVE_SPEC_DEPTHS=2,4 -e VLLM_NVFP4_MLA_PER_TOKEN_SCALE=1 \-e VLLM_ENGINE_READY_TIMEOUT_S=3600 -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \-e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \-e CUTE_DSL_ARCH=sm_121a -e TORCH_CUDA_ARCH_LIST=12.1a -e XDG_CACHE_HOME=/cache/jit \-e NCCL_NET=IB -e NCCL_IB_DISABLE=0 -e NCCL_IB_HCA=$HCA -e NCCL_IB_GID_INDEX=3 \-e NCCL_SOCKET_IFNAME=$NIC -e GLOO_SOCKET_IFNAME=$NIC \-e NCCL_MAX_NCHANNELS=4 -e NCCL_MIN_NCHANNELS=4 -e NCCL_CUMEM_ENABLE=0 \-e RANK=$RANK -e HEAD_IP=$HEAD_IP \aidendle94/sparkrun-vllm-ds4-gb10:production-hybrid-1.3 \bash -lc 'unset VLLM_PREFIX_CACHE_RETENTION_INTERVAL; exec vllm serve /hybridmodel \--served-model-name glm-5.2 --host 0.0.0.0 --port 8210 \--trust-remote-code --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \--enable-prefix-caching \--tensor-parallel-size 4 --decode-context-parallel-size 4 --dcp-comm-backend ag_rs \--attention-backend B12X_MLA_SPARSE \--hf-overrides "{\"index_topk_pattern\":\"FFFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSS\"}" \--kv-cache-dtype nvfp4_ds_mla --kv-cache-memory-bytes 5500000000 \--max-model-len 262144 --max-num-batched-tokens 2048 --max-num-seqs 3 \--gpu-memory-utilization 0.89 \--speculative-config "{\"model\":\"/hybridmodel/mtp-draft\",\"method\":\"mtp\",\"num_speculative_tokens\":4,\"draft_attention_backend\":\"B12X_MLA_SPARSE\",\"adaptive_speculative_tokens_window\":32}" \--compilation-config "{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"custom_ops\":[\"all\"],\"cudagraph_capture_sizes\":[1,2,3,4,5,6,8]}" \--distributed-timeout-seconds 3600 --cpu-distributed-timeout-seconds 3600 \--distributed-executor-backend mp \--nnodes 4 --node-rank $RANK --master-addr $HEAD_IP --master-port 29501'
4. Wait for readiness. First boot loads ~15–20 min per node, then compiles kernels
(the image prewarms all decode shapes so none compile during serving). The /cache/jit
mount makes later boots ~6 min. Ready when curl http://<head>:8210/health returns 200.
bash
curl http://<head>:8210/v1/chat/completions -H "Content-Type: application/json" \-d '{"model":"glm-5.2","messages":[{"role":"user","content":"Capital of Estonia?"}],"max_tokens":64}'
The API is OpenAI-compatible. Reasoning defaults to maximum effort; pass
"chat_template_kwargs":{"enable_thinking":false} to disable thinking or
"reasoning_effort":"high" to lower it.
bash -lc 'unset …; exec vllm serve …' — the image bakes
VLLM_PREFIX_CACHE_RETENTION_INTERVAL (valid for the DeepSeek lineage it also serves),
but GLM-5.2 has no sliding-window KV group, so its KV coordinator rejects any value —
and fails only after the full weight load. Docker -e VAR= sets it to an empty string,
which the parser also rejects (int("")); there is no docker flag to remove a baked
ENV. The only reliable fix is a shell unset before launch, hence the bash -lc
wrapper. exec replaces the shell so SIGTERM reaches vLLM for clean shutdown.RANK/HEAD_IP are referenced inside the single-quoted
bash -lc string, so they must reach the container's environment — they are passed
with (your host shell expands the right-hand side).
/ sit outside the quotes and are expanded by the host shell directly.| Setting | Value | Notes |
|---|---|---|
--kv-cache-memory-bytes | 5.5e9 | 688K-token pool; 7.0e9 → 875K, less host-memory headroom |
--max-num-batched-tokens | 2048 | 812 tok/s prefill; 1024 → 691 tok/s, ~0.5 GiB more headroom |
VLLM_NVFP4_MLA_PER_TOKEN_SCALE | 1 | Per-token KV outer scale (KLD 0.40 vs 4.45 static-calibrated). Keep on; supersedes VLLM_NVFP4_MLA_SCALES_FILE |
VLLM_FP8_LM_HEAD |
On 121 GiB unified-memory nodes, weights + KV + overhead run near capacity. If you use an OOM killer, allow for transient prefill spikes or whitelist the workers.
config.json declares quant_method: "hybrid_mxfp4_ct", a composed quant config
(compressed-tensors for attention/dense, Mxfp4 for the routed experts) unknown to stock
vLLM. Apply the two pieces in vllm_overlay/ — the quant-config class and
an index-authoritative loader filter (see vllm_overlay/INSTALL.md).
MoE then runs through any vLLM MXFP4 backend; only B12X on GB10 (SM121) is tested. The
NVFP4 KV cache (nvfp4_ds_mla) requires a b12x build — with stock vLLM use
--kv-cache-dtype fp8 (656 B/token vs 368, roughly half the KV capacity).
RedHat's attention and dense tensors were extracted byte-exact into hybrid-ct-*.safetensors;
festr2's unchanged MXFP4 expert tensors are referenced by a stitched
model.safetensors.index.json, which is authoritative for tensor-to-file mapping. In v2
the shared experts (layers 3–77) were re-quantized to block-FP8 (128×128, symmetric,
scale = amax/448) directly from the zai-org BF16 reference and swapped into the hybrid-ct
files; the MTP layer-78 shared expert stays BF16. This localized change — the shared expert
is ~5% of the non-expert weights — is the only difference from v1.
The MXFP4 draft measures spec acceptance ~3.0–3.2, versus ~4.4 for a distribution-matched draft on the unmodified base. Testing showed neither a BF16-expert draft nor deeper speculation changes this — the gap comes from the main model's MXFP4 output shift, inherent to the memory trade; the adaptive depth ladder extracts the best decode from it.
zai-org (GLM-5.2, MIT) · RedHat AI (FP8/NVFP4 quantization) · festr2 (MXFP4 expert assembly) · AMD (Quark-calibrated MXFP4 weights). Assembly, loader glue, and GB10 validation by this repo. MIT, as inherited from all sources.
Credits to Koush, David Young, Dooner, Festr, Luke and many others who contribute to the b12x community.
-e RANK=$RANK -e HEAD_IP=$HEAD_IPNICHCAindex_topk_pattern (the 78-char FFFSSS… string) is derived from this config's
indexer_types and must be passed verbatim.mtp-draft/ in this repo is the speculative-decoding draft. To disable speculation,
omit --speculative-config (decode drops to ~12 tok/s).| 0 (optional) |
| opt-in fp8 (W8A16) lm_head: +4-5% decode, tiny logit perturbation (quality-tested: greedy outputs equivalent, GSM8K 96%). Off by default for exact-bf16 logits |
VLLM_ADAPTIVE_SPEC_DEPTHS | 2,4 | Pinning depth 4 measured worse (decode −25%) |
--gpu-memory-utilization | 0.89 | Weights are ~99 GiB/rank; higher risks OOM |