GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
12fn
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
12fn
Model Tree
zai-org/GLM-5.3-Flash
Input Modalities
Output Modalities
Supported Functionality
Requires a vLLM nightly. Stock vLLM 0.30.0 fails on the NoPE cache path (pe_dim must be 64 for fp8_ds_mla). Validated image: vllm/vllm-openai:nightly@sha256:8e6e3752946ff2dde451b313cbaec2ea8e7b52221b21bfbcf53fb68dd4f4c42e (vLLM 0.30.1rc1.dev558, torch 2.13.0+cu130).
bash
vllm serve <this-repo> \ --tensor-parallel-size 2 --dtype bfloat16 --quantization modelopt_mixed \ --moe-backend humming --attention-backend FLASHINFER_MLA_SPARSE_SM120 \ --kv-cache-dtype fp8_ds_mla --kv-cache-memory-bytes 2415919104 \ --gpu-memory-utilization 0.98 --max-model-len 262144 --max-num-seqs 8 \ --disable-custom-all-reduce --no-enforce-eager \ --additional-config '{"kda_prefill_backend":"triton"}' \ --kernel-config '{"enable_flashinfer_autotune":false}' \ --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[1,2,4,8]}' \ --reasoning-parser glm47 --tool-call-parser glm47 --enable-auto-tool-choiceText-only serving skips the vision tower; the weights for it are still included. Use the glm47 parsers, per the official vLLM recipe.
| Concurrency | Output tok/s (aggregate, incl. prefill) | Mean TTFT | Mean TPOT |
|---|---|---|---|
| 1 | 83.2 | 0.45 s | 11.2 ms |
| 4 | 213.7 | 1.35 s | 15.9 ms |
| 8 | 304.0 | 2.11 s | 21.8 ms |
These are spot checks on one machine, not a benchmark-suite comparison against the full-precision model. Upstream benchmark scores on the base model card apply to the original weights, not this quantization.
Released under the MIT License, inherited from the base model. Original model © 2026 Z.AI Co., Ltd. This is an unofficial quantization and is not affiliated with or endorsed by Z.AI.
vllm serve <this-repo> \ --tensor-parallel-size 2 --dtype bfloat16 --quantization modelopt_mixed \ --moe-backend humming --attention-backend FLASHINFER_MLA_SPARSE_SM120 \ --kv-cache-dtype fp8_ds_mla --kv-cache-memory-bytes 2415919104 \ --gpu-memory-utilization 0.98 --max-model-len 262144 --max-num-seqs 8 \ --disable-custom-all-reduce --no-enforce-eager \ --additional-config '{"kda_prefill_backend":"triton"}' \ --kernel-config '{"enable_flashinfer_autotune":false}' \ --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[1,2,4,8]}' \ --reasoning-parser glm47 --tool-call-parser glm47 --enable-auto-tool-choice