GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
JANGQ-AI
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
JANGQ-AI
Model Tree
zai-org/GLM-5.3
Input Modalities
Output Modalities
Supported Functionality
| Module | Source dtype | Bundle dtype |
|---|---|---|
| Routed experts (256 × 3 mats × sparse layers) | BF16 | FP8 E4M3 + F32 weight_scale_inv (block 128×128) |
| Attention (MLA q/kv proj, o_proj) | BF16 | FP8 E4M3 + F32 block scales |
| Dense MLP (first 3 layers) | BF16 | FP8 E4M3 + F32 block scales |
| MTP layer 78 (linear weights) | BF16 | FP8 E4M3 + F32 block scales |
RMSNorms, router gates, e_score_correction_bias | BF16 / F32 | passthrough (BF16 / F32) |
| DSA indexer norms / proj, shared-expert gate | BF16 | passthrough |
embed_tokens, lm_head | BF16 | passthrough |
Dequant per block: w = weight_fp8 * weight_scale_inv, scale = amax / 448.0.
bash
vllm serve JANGQ-AI/GLM-5.3-FP8 \ --tensor-parallel-size 8 \ --trust-remote-codeEnable MTP speculative decoding per vLLM's GLM / DeepSeek MTP docs for a decode speedup (the MTP weights are shipped here).
Converted from the BF16 release with a streaming shard-by-shard quantizer (one
tensor in memory at a time), so no GPU and no large host RAM are needed. The set
of tensors to quantize is taken directly from the official FP8 index rather than
re-derived heuristically, and the output config.json (quantization_config
with modules_to_not_convert) is a verbatim copy of the official FP8 config — so
vLLM treats this identically to the upstream release. The scheme is data-free
(dynamic activation scales), so no calibration set is involved.
Quantized by JANGQ-AI.