GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
RedHatAI
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
RedHatAI
Model Tree
zai-org/GLM-5.3
Input Modalities
Output Modalities
Supported Functionality
This model is a quantized version of zai-org/GLM-5.3, ready for inference with vLLM.
This is a mixed-precision quantization where individual layers are assigned MXFP4 or FP8 precision to achieve an average of ~4.5 bits per parameter while preserving accuracy on sensitive layers.
This model was obtained by quantizing the weights and activations of zai-org/GLM-5.3 to mixed MXFP4 and FP8 precision, ready for inference with vLLM.
Typical MXFP4 quantization pairs MXFP4 weights with MXFP4 activations, a combination that is most performant on Blackwell GPUs. In this model, the majority of the linear layers (routed experts and dense MLPs) instead use MXFP4 weights with FP8 activations quantized dynamically on a per-group basis, making it particularly optimized for inference on Hopper GPUs as of vLLM v0.32. The remaining linear layers (attention, shared experts, and the multi-token prediction layer) are kept at FP8 precision with block-wise weight scaling and FP8 dynamic activations.
This optimization reduces the number of bits per parameter from 16 to approximately 4.5 on average, reducing the disk size and GPU memory requirements by approximately 72% (from ~1507 GB to ~415 GB).
Only the weights and activations of the linear operators within transformer blocks are quantized using LLM Compressor.
text
vllm serve RedHatAI/GLM-5.3-MXFP4xFP8 \ --kv-cache-dtype fp8 \ --tensor-parallel-size 8 \ --speculative-config.method mtp \ --speculative-config.num_speculative_tokens 5 \ --tool-call-parser glm47 \ --reasoning-parser glm47 \ --enable-auto-tool-choiceThis model was created by applying LLM Compressor with a mixed-precision recipe: MXFP4 weight quantization with per-group FP8 dynamic activation quantization for the expert and dense MLP layers, and FP8 block-wise quantization for the attention, shared expert, and multi-token prediction layers. The quantized model was exported in the compressed-tensors format.
Independent reproduction with an open stack: vLLM 0.32.0 (TP=8 on 8xH200, FP8 KV cache) and Inspect AI's ReAct agent, one epoch (pass@1).
| Benchmark | zai-org/GLM-5.3 (FP8, model card) | RedHatAI/GLM-5.3-MXFP4xFP8 |
|---|---|---|
| DeepSWE 1.1 | 66.9 | 67.3 |
| Terminal-Bench 2.1 | 88.2 | 88.8 |
Protocol differences against the model card's setups:
| Model card | This evaluation | |
|---|---|---|
| DeepSWE 1.1 | mini-swe-agent, temperature 0.95, top_p 1.0, 6 h timeout, 400K context | Inspect ReAct, temperature 1.0 / top_p 0.95 (checkpoint defaults), 4 h timeout |
| Terminal-Bench 2.1 | Claude Code 2.1.207, temperature 1.0, top_p 1, 64K max output, 6 h timeout | Inspect ReAct, temperature 1.0 / top_p 0.95, 64K max output per turn, 4 h timeout |
vllm serve RedHatAI/GLM-5.3-MXFP4xFP8 \ --kv-cache-dtype fp8 \ --tensor-parallel-size 8 \ --speculative-config.method mtp \ --speculative-config.num_speculative_tokens 5 \ --tool-call-parser glm47 \ --reasoning-parser glm47 \ --enable-auto-tool-choice