Highlights
- Quality on par with the FP8 release — no measurable degradation (see below).
- ~3.9× smaller than BF16: 388 GB vs. 1.5 TB.
- Runs on A100 (Ampere) — no FP8 hardware needed.
- Serves out of the box with vLLM (
compressed-tensors format).
Evaluation
Evaluated against the reference FP8 deployment of GLM-5.2. Greedy decoding
(temperature 0); reasoning traces stripped and the final answer graded.
Table with columns: Benchmark, This model (W4A16), Reference (FP8)| Benchmark | This model (W4A16) | Reference (FP8) |
|---|
| GSM8K (n=200), exact-match | 96.5% | 94.5% |
| MMLU (n=200), accuracy | 86.5% | 80.0% |
W4A16 matches the FP8 reference within evaluation noise (n=200, standard error
≈ 2 pts). The takeaway is parity — 4-bit quantization retains GLM-5.2's
reasoning and knowledge capability.
Model details
Table | |
|---|
| Base model | zai-org/GLM-5.2 |
| Architecture | GlmMoeDsaForCausalLM (MoE, 78 layers, 256 routed + 1 shared expert, top-8) |
| Weight precision | INT4, group size 128, symmetric |
| Activation precision | BF16 |
| Format | compressed-tensors (pack-quantized) |
| Checkpoint size | 388 GB (8 shards) |
| Context length |
The sparse-attention (DSA) indexer, the MoE router, and the LM head are kept in
BF16; the large linear and expert weights carry the INT4 quantization.
Serving on A100 (8× A100 80 GB, vLLM)
The full INT4 checkpoint fits on one 8×A100-80GB node with room for KV cache.
pip install "vllm>=0.24.0"
# A100 (Ampere) note: use BF16 compute paths and skip Hopper-only kernels.
export VLLM_USE_FLASHINFER_SAMPLER=0 # avoid FlashInfer sampler JIT on some CUDA toolkits
export VLLM_USE_DEEP_GEMM=0 # DeepGEMM (FP8 block-scale) is not needed on A100
vllm serve lowbitcoffee/GLM-5.2-W4A16 \
--tensor-parallel-size 8 \
--dtype bfloat16 \
--max-model-len 32768 \
--gpu-memory-utilization 0.92 \
--served-model-name glm-5.2-w4a16 \
--trust-remote-code
vLLM auto-detects the quantization from the checkpoint — no --quantization
flag is required. Increase --max-model-len toward the model's 1M limit only if
you have KV-cache headroom; lower it to raise concurrency.
On 8× A100 40 GB, the weights alone (388 GB) exceed the 320 GB of aggregate
VRAM — use two nodes (--tensor-parallel-size 16) or the 80 GB SKU.
Query it (OpenAI-compatible)
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "glm-5.2-w4a16",
"messages": [{"role": "user", "content": "What is 84 * 3 / 2?"}],
"max_tokens": 1024,
"temperature": 0
}'
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
model="glm-5.2-w4a16",
messages=[{"role": "user", "content": "Explain MoE routing in two sentences."}],
max_tokens=1024,
temperature=0.6,
)
print(resp.choices[0].message.content)
GLM-5.2 is a reasoning model: responses may include a <think>…</think> block
before the final answer. Strip it client-side, or configure a reasoning parser
in your serving stack if you want the fields separated.
License
Released under the MIT license, inheriting the license of the base model
zai-org/GLM-5.2.