c-bf

GLM-5.2-AutoRound-W4G64-MTP

Checkpoint summary

Table with columns: Property, Value
Property	Value
Base model	`zai-org/GLM-5.2`
Architecture	`GlmMoeDsaForCausalLM`
Quantization	AutoRound W4A16, symmetric W4G64
Packing	`auto_round:auto_gptq`
Checkpoint layout	81 backbone shards + 5 MTP shards
Approximate storage	405 GiB
MTP	One quantized next-token-prediction layer
License	MIT, inherited from the base model

Quantization procedure

This is post-training weight-only quantization; the model was not fine-tuned. The run used AutoRound 0.14.0 on four NVIDIA H20 GPUs with:

Table with columns: Setting, Value
Setting	Value
Weight bits	4
Group size	64
Symmetric	Yes
Optimization iterations	200
Calibration samples	512
Calibration sequence length	2,048
Batch size	2
Gradient accumulation	4
Scale dtype	FP16

The calibration set consisted of packed coding-agent-style text samples. The calibration corpus is not distributed with this checkpoint.

The mixed-precision policy was intentionally conservative:

Backbone layers 3–77: routed-expert gate_proj, up_proj, and down_proj matrices are W4G64.
Backbone layers 0–2, all attention and indexer modules, shared experts, eh_proj, weights_proj, embeddings, and lm_head remain BF16.
MTP layer 78 was processed separately: 768 routed-expert weights are W4G64; 23 non-expert tensors remain BF16. The MTP shared head reuses the BF16 target head at runtime.

Validated runtime

The tested software stack was:

Table with columns: Component, Version
Component	Version
vLLM	`0.23.1rc1.dev471+ge312c5cb2`
vLLM commit	`e312c5cb25427e76fc3830ab14e7b6bc0963a55c`
Python	3.12.12
PyTorch	2.11.0+cu130
CUDA	13.0
Hardware	8 × NVIDIA H20 96 GB, tensor parallel size 8

apply_vllm_glm52_patches.py contains the source-level compatibility changes used for this checkpoint. It is strictly hash-pinned to the vLLM commit and Triton source above, stages all edits before installation, compiles and verifies every output, installs files atomically, and is idempotent. It refuses unknown or partially modified sources instead of attempting a fuzzy patch.

The patch covers:

GLM-5.2 sparse-indexer and missing-parameter guards, plus AutoRound routed expert and MTP checkpoint namespace compatibility.
Header-first safetensors filtering so MTP eager loading does not read every unrelated backbone shard payload.
CUDA Graph pre-capture warmup and real cubin generation through ptxas.
Explicit PyNCCL loader/device handling and correct CUDA-device binding for the CPU KV-cache pinning thread.

These changes affect loading and serving only; they do not modify model weights. Use a separate environment for this pinned runtime. A newer vLLM version may already contain equivalent fixes and should be validated independently.

vLLM serving example

The following is a public, minimal reproduction of the tested serving profile. It assumes that vLLM has already been built at the commit above and that ptxas is available on PATH. Replace the example paths and public model name.

bash
export VENV=/path/to/vllm-venv
export MODEL_DIR=/path/to/GLM-5.2-AutoRound-W4G64

"$VENV/bin/python" "$MODEL_DIR/apply_vllm_glm52_patches.py" \
  --venv-root "$VENV" \
  --model-dir "$MODEL_DIR" \
  --apply

# Keep compilation and CUDA Graphs enabled. These settings match the validated
# single-node runtime; adapt NCCL transport selection to your topology.
export VLLM_DISABLE_COMPILE_CACHE=1
export VLLM_DISABLE_PYNCCL=0
export NCCL_CUMEM_ENABLE=0
export NCCL_CUMEM_HOST_ENABLE=0
export VLLM_ALLREDUCE_USE_SYMM_MEM=0
export VLLM_USE_NCCL_SYMM_MEM=0
export TRITON_STORE_BINARY_ONLY=0
export CUDA_MODULE_LOADING=EAGER

"$VENV/bin/vllm" serve "$MODEL_DIR" \
  --served-model-name glm-5.2-autoround-w4g64 \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 8 \
  --dtype auto \
  --max-model-len 130000 \
  --gpu-memory-utilization 0.94 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 8192 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --chat-template-content-format string \
  --trust-remote-code \
  --safetensors-load-strategy eager \
  --distributed-timeout-seconds 1800 \
  --disable-custom-all-reduce \
  --enable-prefix-caching \
  --compilation-config.pass_config.fuse_allreduce_rms false \
  --speculative-config '{"method":"mtp"}' \
  --spec-tokens 2 \
  --kv-offloading-size 256 \
  --kv-offloading-backend native \
  --disable-hybrid-kv-cache-manager

The 256 GiB CPU KV offload setting is optional and requires sufficient pinned host memory. Reduce or remove it for systems without that capacity.

Example OpenAI-compatible request:

bash
curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "glm-5.2-autoround-w4g64",
    "messages": [{"role": "user", "content": "Explain and solve x^2 = 1 (mod 105)."}],
    "max_tokens": 8192,
    "temperature": 0.6,
    "top_p": 0.95,
    "stream": true
  }'

Reasoning models may use several thousand tokens before final content. For long-horizon coding-agent workloads, a 32K output allowance was materially more reliable than an 8K allowance.

Measured performance

Measurements below used the validated stack on one 8 × H20 node with TP8, 130K maximum context, MTP speculative depth 2, max_num_seqs=8, an 8,192-token batch ceiling, prefix caching, CUDA Graphs, and 256 GiB native CPU KV offload.

The fixed-shape serving test used the same 117-token prompt, exactly 512 output tokens per request, streaming responses, and a 60-second load window. Throughput is the server generation-token counter divided by wall time.

Table with columns: Concurrency, Successful requests, Aggregate generation, Mean / P95 TTFT, Mean prefill, Mean decode
Concurrency	Successful requests	Aggregate generation	Mean / P95 TTFT	Mean prefill	Mean decode
3	30 / 30	229.09 tok/s	0.411 / 0.859 s	0.351 s	6.116 s
5	40 / 40	310.30 tok/s	0.438 / 0.679 s	0.361 s	7.586 s
8	56 / 56

No request preemption or metrics scrape error occurred in these windows. MTP accepted 57.7%–58.7% of drafted tokens, for an effective accepted length of about 2.16 tokens per verification iteration.

A separate reasoning smoke test produced 4,083 tokens at 110.16 tok/s after a 0.266-second TTFT, stopped normally, returned both reasoning and final content, and correctly enumerated the eight solutions of x² ≡ 1 (mod 105).

On isolated coding-agent tasks, the model passed all public and held-out tests for the four completed single-concurrency tasks, ranging from configuration merging to a parser/formatter/evaluator. In concurrent agent runs, aggregate generation was 218.20 tok/s at concurrency 3 and 278.04–292.79 tok/s at concurrency 5. These are workload observations, not standardized leaderboard results.

Performance varies with prompt length, output distribution, MTP acceptance, cache state, host-memory bandwidth, software build, and GPU topology. The numbers above should not be compared with other model cards unless the harness and serving configuration are matched.

Quality and limitations

This checkpoint has not been evaluated against the BF16 base model with a comprehensive, identical public benchmark harness. No claim of lossless quantization is made.
Only the vLLM commit and 8 × H20 configuration documented above were fully validated. Other vLLM revisions, runtimes, GPU architectures, and tensor parallel sizes may require additional work.
Quantization is concentrated in routed experts, but quality can still differ from the BF16 base model, especially on rare domains or long reasoning paths.
The checkpoint inherits the capabilities, risks, usage restrictions, and limitations described in the official GLM-5.2 model card.
Users are responsible for application-specific safety, privacy, reliability, and output-quality evaluation before deployment.

License and references

This quantized checkpoint follows the base model's MIT license. Review the base repository and license before redistribution or production use.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

c-bf

Model Tree

Base

zai-org/GLM-5.2

Quantized

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Container

Explore FriendliAI today

Get started Talk to an engineer

Checkpoint summary

Table with columns: Property, Value
Property	Value
Base model	`zai-org/GLM-5.2`
Architecture	`GlmMoeDsaForCausalLM`
Quantization	AutoRound W4A16, symmetric W4G64
Packing	`auto_round:auto_gptq`
Checkpoint layout	81 backbone shards + 5 MTP shards
Approximate storage	405 GiB
MTP	One quantized next-token-prediction layer
License	MIT, inherited from the base model

Quantization procedure

This is post-training weight-only quantization; the model was not fine-tuned. The run used AutoRound 0.14.0 on four NVIDIA H20 GPUs with:

Table with columns: Setting, Value
Setting	Value
Weight bits	4
Group size	64
Symmetric	Yes
Optimization iterations	200
Calibration samples	512
Calibration sequence length	2,048
Batch size	2
Gradient accumulation	4
Scale dtype	FP16

The calibration set consisted of packed coding-agent-style text samples. The calibration corpus is not distributed with this checkpoint.

The mixed-precision policy was intentionally conservative:

Backbone layers 3–77: routed-expert gate_proj, up_proj, and down_proj matrices are W4G64.
Backbone layers 0–2, all attention and indexer modules, shared experts, eh_proj, weights_proj, embeddings, and lm_head remain BF16.
MTP layer 78 was processed separately: 768 routed-expert weights are W4G64; 23 non-expert tensors remain BF16. The MTP shared head reuses the BF16 target head at runtime.

Validated runtime

The tested software stack was:

Table with columns: Component, Version
Component	Version
vLLM	`0.23.1rc1.dev471+ge312c5cb2`
vLLM commit	`e312c5cb25427e76fc3830ab14e7b6bc0963a55c`
Python	3.12.12
PyTorch	2.11.0+cu130
CUDA	13.0
Hardware	8 × NVIDIA H20 96 GB, tensor parallel size 8

The patch covers:

GLM-5.2 sparse-indexer and missing-parameter guards, plus AutoRound routed expert and MTP checkpoint namespace compatibility.
Header-first safetensors filtering so MTP eager loading does not read every unrelated backbone shard payload.
CUDA Graph pre-capture warmup and real cubin generation through ptxas.
Explicit PyNCCL loader/device handling and correct CUDA-device binding for the CPU KV-cache pinning thread.

vLLM serving example

bash
export VENV=/path/to/vllm-venv
export MODEL_DIR=/path/to/GLM-5.2-AutoRound-W4G64

"$VENV/bin/python" "$MODEL_DIR/apply_vllm_glm52_patches.py" \
  --venv-root "$VENV" \
  --model-dir "$MODEL_DIR" \
  --apply

# Keep compilation and CUDA Graphs enabled. These settings match the validated
# single-node runtime; adapt NCCL transport selection to your topology.
export VLLM_DISABLE_COMPILE_CACHE=1
export VLLM_DISABLE_PYNCCL=0
export NCCL_CUMEM_ENABLE=0
export NCCL_CUMEM_HOST_ENABLE=0
export VLLM_ALLREDUCE_USE_SYMM_MEM=0
export VLLM_USE_NCCL_SYMM_MEM=0
export TRITON_STORE_BINARY_ONLY=0
export CUDA_MODULE_LOADING=EAGER

"$VENV/bin/vllm" serve "$MODEL_DIR" \
  --served-model-name glm-5.2-autoround-w4g64 \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 8 \
  --dtype auto \
  --max-model-len 130000 \
  --gpu-memory-utilization 0.94 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 8192 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --chat-template-content-format string \
  --trust-remote-code \
  --safetensors-load-strategy eager \
  --distributed-timeout-seconds 1800 \
  --disable-custom-all-reduce \
  --enable-prefix-caching \
  --compilation-config.pass_config.fuse_allreduce_rms false \
  --speculative-config '{"method":"mtp"}' \
  --spec-tokens 2 \
  --kv-offloading-size 256 \
  --kv-offloading-backend native \
  --disable-hybrid-kv-cache-manager

The 256 GiB CPU KV offload setting is optional and requires sufficient pinned host memory. Reduce or remove it for systems without that capacity.

Example OpenAI-compatible request:

bash
curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "glm-5.2-autoround-w4g64",
    "messages": [{"role": "user", "content": "Explain and solve x^2 = 1 (mod 105)."}],
    "max_tokens": 8192,
    "temperature": 0.6,
    "top_p": 0.95,
    "stream": true
  }'

Reasoning models may use several thousand tokens before final content. For long-horizon coding-agent workloads, a 32K output allowance was materially more reliable than an 8K allowance.

Measured performance

Table with columns: Concurrency, Successful requests, Aggregate generation, Mean / P95 TTFT, Mean prefill, Mean decode
Concurrency	Successful requests	Aggregate generation	Mean / P95 TTFT	Mean prefill	Mean decode
3	30 / 30	229.09 tok/s	0.411 / 0.859 s	0.351 s	6.116 s
5	40 / 40	310.30 tok/s	0.438 / 0.679 s	0.361 s	7.586 s
8	56 / 56

Quality and limitations

This checkpoint has not been evaluated against the BF16 base model with a comprehensive, identical public benchmark harness. No claim of lossless quantization is made.
Only the vLLM commit and 8 × H20 configuration documented above were fully validated. Other vLLM revisions, runtimes, GPU architectures, and tensor parallel sizes may require additional work.
Quantization is concentrated in routed experts, but quality can still differ from the BF16 base model, especially on rare domains or long reasoning paths.
The checkpoint inherits the capabilities, risks, usage restrictions, and limitations described in the official GLM-5.2 model card.
Users are responsible for application-specific safety, privacy, reliability, and output-quality evaluation before deployment.

License and references

This quantized checkpoint follows the base model's MIT license. Review the base repository and license before redistribution or production use.

GLM-5.2-AutoRound-W4G64-MTP

README

Checkpoint summary

Quantization procedure

Validated runtime

vLLM serving example

Measured performance

Quality and limitations

License and references

Explore FriendliAI today

README

Checkpoint summary

Quantization procedure

Validated runtime

vLLM serving example

Measured performance

Quality and limitations

License and references