GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
brandonmusic
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
brandonmusic
Model Tree
zai-org/GLM-5.3-Flash
Input Modalities
Output Modalities
Supported Functionality
The dedicated runtime image is published at:
text
verdictai/glm53-flash-exl3-k4:r19-sm120-tp2-v44verdictai/glm53-flash-exl3-k4:daily-driver-sm120-tp2OCI index: sha256:15192e3930b4ae5558271ebe7d1a5a02da6dcc5a6c292c44e79a3fb8c883b5e1amd64 manifest: sha256:0c35421c2c773743ee74b17592d5bd32143546a2bbfe32fc8f32b97ca74167bf
The checkpoint stores global unsliced 2,048-wide routed experts. During load, the runtime streams the correct half directly into each rank's 1,024-wide TP2 B12X slab. All 288 routed experts stay on the fused EXL3/Trellis K4 path; they are not reconstructed as BF16.
The port implements the released 45-layer hybrid architecture: 34 KDA
linear-attention layers, 11 sparse NoPE MLA layers, IndexPool-4 compression,
and top-k 2,048 sparse selection. FP8 MLA KV uses
FLASHINFER_MLA_SPARSE_SM120; native NVFP4 MLA KV uses the
B12X_MLA_SPARSE GLM-NoPE path. Image v44 fixes mixed-MTP graph padding in
DCP2 before sparse scoring and KV writes.
The image contains the runtime only. Download or mount this model separately.
Download the published launch files:
bash
hf download brandonmusic/GLM-5.3-Flash-EXL3-4bpw \compose.yaml serve-glm53-sm120-tp2.sh runtime-results-v44.json \--local-dir glm53-runtimecd glm53-runtime
The Compose default is the accuracy-first profile: TP2, DCP2, FP8 MLA KV, CUDA graphs, MTP3, and a measured 435,456-token maximum context.
bash
export GLM53_MODEL_PATH=/absolute/path/to/GLM-5.3-Flash-EXL3-4bpwexport GLM53_GPU_0=0 GLM53_GPU_1=1 GLM53_PORT=8012docker compose pulldocker compose up -ddocker compose logs -f
Equivalent serve-script launch:
bash
chmod +x serve-glm53-sm120-tp2.shMODEL=/absolute/path/to/GLM-5.3-Flash-EXL3-4bpw \GPU_DEVICES=0,1 PORT=8012 \./serve-glm53-sm120-tp2.sh
For the qualified 499K profile, use native NVFP4 MLA KV:
bash
PROFILE=long500k MODEL=/absolute/path/to/GLM-5.3-Flash-EXL3-4bpw \GPU_DEVICES=0,1 PORT=8012 \./serve-glm53-sm120-tp2.sh
TP is fixed at two. DCP=1 remains available for controlled comparisons;
the published defaults use DCP2. The generation configuration uses the
released temperature 1.0 and top-p 0.95. The server exposes vLLM's glm45
reasoning parser.
Example request:
bash
curl http://127.0.0.1:8012/v1/chat/completions \-H 'Content-Type: application/json' \-d '{"model":"GLM-5.3-Flash-EXL3-4bpw","messages":[{"role":"user","content":"Explain sparse attention briefly."}],"temperature":1.0,"top_p":0.95}'
For bounded structured-analysis requests similar to LAVD, setting
"reasoning_effort":"low" prevents the reasoning trace from consuming the
entire output budget; that is the qualified 10-run LAVD regime.
Stable TP2/DCP2/CUDA-graph/MTP3/NVFP4 results, prefix caching disabled:
| Context | Prefill | C1 decode |
|---|---|---|
| 0 | — | 98.9 tok/s |
| 8K | 3,819 tok/s | — |
| 16K | 4,112 tok/s | 106.5 tok/s |
| 32K | 4,149 tok/s | 101.2 tok/s |
| 64K | 4,174 tok/s | 107.2 tok/s |
| 128K | 4,145 tok/s | 112.4 tok/s |
Public-digest FP8 validation measured C1 decode at 121.4, 120.8, and 109.5 tok/s at 0, 16K, and 32K. Its cold prefill scout measured 3,414, 3,550, 3,580, and 2,524 tok/s at 8K, 16K, 32K, and 64K.
Fused-kernel block 32 improves the 1,024-token scheduler regime, but block 64 is faster with the published 2,048-token prefill chunk. The best stable long-context prefill measured here is about 4.1K tok/s; no 10K result is claimed.
Only the required sealed 2,048-token teacher window was downloaded from the BF16-logit dataset. Every run below scores all 2,047 causal positions.
| MLA KV cache | Runtime regime | Five-run mean KLD | Population stddev | Mean top-1 | Gate |
|---|---|---|---|---|---|
| FP8 | TP2, DCP1, eager, no MTP | 0.0246285766 | 0.0003261567 | 0.9379580 | pass |
| NVFP4 | TP2, DCP1, eager, no MTP | 0.0605348505 | 0.0 | 0.9154861 | fail by 0.0005348505 |
The preregistered gate is mean KLD <0.06. The FP8 result passes in every
run. NVFP4 narrowly misses and is not described as an accuracy pass. The
first 64 rows disproportionately affect both regimes, but are retained; the
measurement is over the full causal window rather than a trimmed subset.
Teacher payload SHA-256:
9f49af1b1b1a6ac88a00f5feaa89c25232597306d73d7c0cd30bb7e9c775cfb6.
All quality requests use temperature 1.0 and top-p 0.95.
| Test | Regime | Result |
|---|---|---|
| Estonia, 10 runs | TP2/DCP2/MTP3/NVFP4 | 10/10 correct, no cap or request errors |
| LAVD, 10 runs | TP2/DCP2/MTP3/FP8, low reasoning | independent audit: 1 exact + 9 near = 10/10 accepted; no caps/errors |
| Needle through 499K | TP2/DCP2/MTP3/NVFP4 | raw 17/18; identical final prompt retry recovered exact needle |
The LAVD harness's original numeric extractor reported 6 near and 4 failures. A SHA-bound independent audit of the preserved output finds the explicit answer line and scores 1 exact plus 9 near. The false failures came from incidental numbers in explanatory prose. Average LAVD generation throughput was 160.0 tok/s.
The only raw needle miss occurred at 499K/depth 0.9 after the model consumed
the exact 1,600 tokens remaining under max-model-len=499968 without reaching
final content. The identical 498,368-token prompt with low reasoning returned
the exact secret in seven tokens. This is retained as 17/18 plus a targeted
pass, not rewritten as an 18/18 raw matrix.
The v44 NVFP4 DCP2 graph profile allocated 608,656 KV tokens and served a 499,968-token maximum context with 1.22x headroom. A qualified smaller-chunk run allocated 678,968 tokens. FP8 stores wider cache records, so the daily profile uses the measured 435,456-token ceiling.
Hotel is omitted because the owner explicitly stopped that test.
The full teacher dataset is
brandonmusic/GLM-5.3-Flash-BF16-Teacher-Logits.
The runtime qualification downloaded only the required window, not the full
843 GB payload.
Image provenance labels bind vLLM base
df684ff47dcbf088b41311494fd20347a702e56a and B12X GLM branch
30aafad96b7a78064651c4a5ac177791e7bdee30. Exact benchmark and source hashes
are included in runtime-results-v44.json.