GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
djdeniro
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
djdeniro
Model Tree
zai-org/GLM-5.3-Flash
Input Modalities
Output Modalities
Supported Functionality
This is a composite-quantized checkpoint of Z.ai's GLM-5.3-Flash — the first natively
multimodal model in the GLM-5 series (321B total / ~18B active parameters). It was quantized from
the official FP8 release using the tcclaviger/vllm composite quantizer and is intended to be
served with vLLM on RDNA4 (AMD Radeon R9700) hardware.
📦 Companion serving repo: GLM-5.3-Flash-rocm-r9700 — the ROCm/RDNA4 overlay + quantization recipe.
| Configuration | GPQA Diamond | Correct | Empty | Answered |
|---|---|---|---|---|
| GLM-5.3-Flash (Z-AI API) | 80.8% | 76 | 6 | 94 |
| GLM-5.3-Flash-RFA-RFI8 | 85.1% | 80 | 6 | 94 |
Total Context Limit for each task in test 32k, means 6x tasks use more than 32k output tokens
| Scenario | Throughput |
|---|---|
| Decode, batch size 1, MTP OFF | ~34–39 tok/s |
| Decode, batch size 1, MTP spec=3 | ~82–88 tok/s |
| Aggregate, 4 concurrent, MTP spec=3 | ~155 tok/s |
| Context window (fp8 KV) | 300,000 tokens |
MTP speculative decoding: mean acceptance length ~3.7–3.9 of 4 draft tokens, average draft acceptance 91–97% (live engine metrics, GPQA-style prompts).
| Component | Credit |
|---|---|
| Base model | zai-org/GLM-5.3-Flash (Z.ai, MIT) |
| Quantizer & quant kernels (RFA / RFI) | tcclaviger/vllm:latest (IronLLM Labs) |
| RDNA4 port + quant recipe | GLM-5.3-Flash-rocm-r9700 (this project's overlay) |
The composite RFA + RFI8 quantization scheme, its kernels, and the serving runtime are provided
by tcclaviger/vllm (IronLLM Labs). This checkpoint is the result of applying a quantization
recipe (archspec + source patches + kda-remap, in the companion repo) on top of that stack, and a
port of the GLM-5.3-Flash architecture to the RDNA4 serving path.
The checkpoint mixes three schemes in one quant_method: "rfi" composite:
| Scheme | Bits | Applied to | Stored size |
|---|---|---|---|
| RFA | 4.5 bpw | MoE routed experts (42 layers × 288 experts) | ~171.3 GB |
| RFI8 | 8 bpw | attention / shared-expert / dense linears | ~7.8 GB |
| BF16 / FP32 | 16 / 32 bpw | embeddings, vision tower, norms, MTP layer | ~18.7 GB |
markdown
bpw = (total safetensors bytes × 8) / total parameters= (197,843,715,288 × 8) / 321,342,220,638= 4.9254 ≈ 4.93 bpw
| Metric | Value |
|---|---|
| Total parameters | 321,342,220,638 (~321.3B) |
| On-disk size | 197.8 GB · 25 safetensors shards |
| Average bpw | 4.9254 ≈ 4.93 |
| vs. FP8 source | 0.60× (197.8 GB vs 328.3 GB) |
| vs. BF16 | 0.31× (197.8 GB vs 642.7 GB) |
bash
docker pull tcclaviger/vllm:latestgit clone https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700 overlay# current production config (MTP spec=3, fp8 KV, 300k context)docker run --rm --tty --ipc=host --shm-size=128g \--device /dev/kfd:/dev/kfd --device /dev/dri:/dev/dri \-v /path/to/GLM-5.3-Flash-RFA-RFI8-8xR9700:/models:ro \-v "$PWD/overlay":/overlay:ro \--entrypoint bash tcclaviger/vllm:latest \-c "/overlay/apply_overlay.sh && GLM5_NEXT_MTP_PROPOSER=1 exec vllm serve /models \--served-model-name glm53-flash --trust-remote-code --quantization rfi \--tensor-parallel-size 8 --gpu-memory-utilization 0.9575 \--max-model-len 300000 --max-num-seqs 4 --max-num-batched-tokens 2048 \--kv-cache-dtype fp8 \--speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}' \--enable-prefix-caching --distributed-executor-backend mp \--compilation-config '{\"cudagraph_capture_sizes\":[1,2,4,8,16],\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_copy_inputs\":true}'"
| Property | Value |
|---|---|
| Architecture | Glm5NextForConditionalGeneration |
| Layers | 45 = 34 KDA (linear attention) + 11 DSA (sparse-MLA) |
| Routed experts | 288 (top-8) + 1 shared expert |
| Extra | mHC hyper-connections, 1 nextn MTP draft layer, native vision tower |
| Context (fp8 KV) | 300,000 tokens |
Images are resized with aspect ratio preserved, clamped to min 384×384 / max 1280×1280, and fed with a min/max image-token budget. The model accepts image and video inputs natively.
--kv-cache-dtype fp8 and scales fixed
at 1.0. Do not enable --calculate-kv-scales: runtime calibration on the profile dummy-run
produces garbage scales from the uninitialized KDA recurrent state (details in the
serving repo).reasoning_effort="low" — the default Reasoning Effort Max spends 16k+ tokens
thinking before producing content on long generations.MIT. Base model © Z.ai (zai-org), MIT license. Quantizer & runtime © IronLLM Labs (tcclaviger/vllm).