GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
cyijun2k
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
cyijun2k
Model Tree
LibertAIDAI/GLM-5.3-Flash-NVFP4
Input Modalities
Output Modalities
Supported Functionality
The fixture keeps dimensions that select execution branches, kernels, packed weight layouts, or cache ABIs:
| Contract | Fixture and target value |
|---|---|
| layer pattern | 3 KDA + 1 sparse MLA |
| attention / KV heads | 64 / 64 |
| Q LoRA / KV LoRA rank | 1536 / 512 |
| QK NoPE / RoPE dimension | 256 / 0 |
| V head dimension | 256 |
| KDA heads / head dimension / convolution | 64 / 128 / 4 |
| index heads / head dimension | 32 / 128 |
| index top-k / k-pool | 2048 / 4 |
| routed / selected / shared experts | 288 / 8 / 1 |
| MoE intermediate size | 2048 |
| dense MLP intermediate size | 12288 |
| dense prefix | first 3 layers |
| MTP layers | 1 |
| vocabulary and tokenizer | 154880 |
The routed-expert tensors use the same ModelOpt NVFP4 storage ABI as the source
checkpoint: packed U8 weights, F8_E4M3 per-block scales, and scalar F32
global scales. All 1,728 expert projections are present.
To keep the artifact under 1 GiB, hidden_size is reduced from 4096 to 256,
the 45-layer stack is reduced to its first four-layer cycle, maximum positions
are reduced to 8192, and the vision tower is reduced to one small layer. The
four base layers plus one copied MTP layer still cover KDA, dense MLP, sparse
MLA, routed NVFP4 MoE, shared experts, and speculative decoding.
The pinned base image
vllm/vllm-openai:glm53-flash-arm64-cu130@sha256:905c02933be6021301db2dc284e24e3727467aa3a0f63b41d609885778a07bce
loads the checkpoint but needs three compatibility changes:
qk_rope_head_dim=0), while the SM120
fp8_ds_mla cache writer requires a physical 64-BF16 positional region.
The adapter writes zeros into that region and pads the absorbed 512-wide
query to 576. This preserves the NoPE dot product.index_topk=2048, index_kpool=4, the always-selected tail, and
BLOCK_N=128 alignment produce a physical sparse-index buffer of 2176
columns. The bundled FlashInfer AOT module only instantiates 2048. The
adapter adds the exact num_heads=64, topk=2176 decode specialization.GLM_NSA, num_heads=64, topk=2176 prefill specialization is also
required. The patched module is rebuilt for sm_121a; editing Python or JIT
source alone does not override the precompiled AOT .so in the base image.The physical sparse-MLA row remains the native GLM_NSA 656-byte layout:
text
query = [absorbed NoPE 512 | zero padding 64]KV = [FP8 latent 512 | four FP32 scales | 64 BF16 zeros]
The vLLM CLI still requires --block-size 256 because the GLM index cache must
be divisible by index_kpool * 32. Internally the compressed FlashInfer sparse
cache uses 64-token physical pages; these are different levels of the cache
layout and should not be conflated.
bash
docker run --rm --gpus all --ipc=host -p 8000:8000 \ghcr.io/cyijun/glm-5.3-flash-nvfp4-gb10:vllm-glm53-sm121 \cyijun2k/glm-5.3-flash-tiny-random-nvfp4 \--served-model-name glm53-tiny \--max-model-len 512 \--max-num-seqs 2 \--gpu-memory-utilization 0.10 \--block-size 256 \--moe-backend marlin \--enforce-eager
Optional one-token MTP smoke test:
text
--speculative-config '{"method":"mtp","num_speculative_tokens":1}'
Validation was performed on an NVIDIA GB10 (SM121) with vLLM
0.1.dev20051+g487ecf187, FlashInfer 0.6.17, Transformers 5.15.1, and
PyTorch 2.13.0+cu130.
| Check | Result |
|---|---|
| critical configuration fields against pinned source revision | 21/21 match |
| fixture tensor names with an analogous source tensor | 5342/5342 |
| NVFP4 weight/scale/global-scale dtypes | U8 / F8_E4M3 / F32 |
model load and /health | pass |
| OpenAI-compatible short chat + continuous decode | pass |
| 491-token prefill + 16-token decode | pass |
See VALIDATION.json for the exact image digest and machine-readable results.
model.safetensors is 1,010,842,392 bytes and contains 5,342 tensors.
text
sha256 d0f8a670c6d96be8984db205a4bfb32c947f09182d6b327fab8d4f1088b66810
The checkpoint is deterministic with model seed 42 and expert payload seed
4242. BUILD_INFO.json records the source revision and fixture profile.
| MTP with one speculative token | pass |
| public companion GHCR manifest | sha256:4251b561d111d817765ed4097512ce36811deac071a4a7411d20242df5c74a47 |