GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
tacos4me
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
tacos4me
Model Tree
LibertAIDAI/GLM-5.3-Flash-NVFP4
Input Modalities
Output Modalities
Supported Functionality
| 512k "everything" lane | 262k MTP lane | old offloaded setup¹ | |
|---|---|---|---|
| max context | 524,288 | 262,144 | 262,144 |
| CPU offload | none | none | 6 GiB/rank |
| decode, short ctx | 144.5 tok/s | 129–146 tok/s | 87 tok/s |
| speculative decode (MTP) | k=1 (95.8% accept) | k=1 | k=1 |
| multimodal | vision on — 10/10 spot-set | text-only² | yes |
| largest served prompt | 501,093 tok (3/3 clean, 90–96 s) | 34k tested | — |
| quality (TF top-1 vs BF16 base) | **94.10%**³ | 95.57% | reference |
¹ Different feature set (BF16 attention + offloaded experts); context, not baseline.
² Vision tower ships unquantized; not loaded in this lane.
³ The fp6 compact-KV rung (VLLM_KVQ_TILES=6666) is what fits MTP + vision KV at 524,288 —
pool 548,560 tokens from a 3.07 GiB/rank pin, min-free watermark 270 MiB — and costs ~1.5 TF
points for it. The same checkpoint with text-only 512k flags measures 95.57% (gate ≥95) at
101.7 tok/s — see the max-quality variant in Quickstart. Repetition probes at 251k: clean
(worst 4-gram repeat 3 vs fail ≥12).
[!IMPORTANT] Matched pair: this checkpoint does not load on stock vLLM or the vendor images.
glm5_nextlives in per-model images (vllm#53906 still open) which construct attention unquantized and cannot load an FP8lm_head. Use the patch set inserving/(Dockerfile included). Without it:KeyErroronweight_scale_invor a vocab assert.
bash
docker build -t local/vllm-glm53:fp8attn-512k serving/docker build -t local/vllm-glm53:everything serving/everything-lane/
The 512k "everything" lane (MTP + vision + 524,288, the headline config):
bash
docker run --init --rm --gpus all --runtime nvidia --ipc=host --network host \--shm-size=32g --ulimit memlock=-1 --ulimit stack=67108864 \-e VLLM_ENGINE_READY_TIMEOUT_S=3600 \-e VLLM_KVQ_TILES=6666 \-v /path/to/GLM-5.3-Flash-NVFP4-FP8ATTN-512K:/model:ro \local/vllm-glm53:everything /model \--served-model-name glm-5.3-flash \--host 0.0.0.0 --port 8000 \--tensor-parallel-size 2 \--max-model-len 524288 \--gpu-memory-utilization 0.95 \--kv-cache-memory 3300000000 \--kv-cache-dtype fp8 \--max-num-seqs 1 \--max-num-batched-tokens 1024 \--limit-mm-per-prompt '{"image":1,"video":0}' \--speculative-config '{"method":"mtp","num_speculative_tokens":1}' \--kernel-config '{"enable_jit_warmup":false,"enable_cutedsl_warmup":false}' \--compilation-config '{"cudagraph_capture_sizes":[1,2]}' \--enable-prefix-caching \--no-enable-flashinfer-autotune \--enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm45 \--trust-remote-code
Capture size 2 matters: MTP decode steps are 2 tokens — with only [1] captured, decode runs
eager at ~27 tok/s instead of ~144. Images go in OpenAI image_url content parts (data URLs
supported). VLLM_KVQ_TILES controls the compact-KV rung: 6666 (fp6, the only rung that fits 512k —
measured 5.87 KiB/tok effective), 8888 (bit-identical repack, ~25% more KV/token — trades
context ceiling for zero quality delta), unset (stock rows; 327,680 is the ceiling for this
feature set, measured pool 417,631 tokens).
Max-quality text-only variant at 512k (TF 95.57% vs BF16 base, 101.7 tok/s) — image
local/vllm-glm53:fp8attn-512k, drop -e VLLM_KVQ_TILES, --speculative-config and vision:
markdown
--kv-cache-memory 4241026048 --limit-mm-per-prompt '{"image":0,"video":0}' --compilation-config '{"cudagraph_capture_sizes":[1]}'
262k MTP speed lane — the text-only command with:
markdown
--max-model-len 262144 --kv-cache-memory 3113851289 --max-num-seqs 2
--compilation-config, add
--speculative-config '{"method":"mtp","num_speculative_tokens":1}'. Measured 129–146 tok/s
single-stream (MTP acceptance is content-dependent).
Margin disclosure: 270–290 MiB min-free watermark during a full 512k prefill (everything / text-only lane respectively), with recoverable allocator retries by design — zero failures across both 3-prompt soaks. Do not co-locate other GPU tenants during long prefills.
--kv-cache-memory (pinned) — the load-bearing trick, and a disclosed trade: the pin bypasses
vLLM's memory-profiling gate. On this GPU the boot-time free-memory reading is always ~93.89 of
94.97 GiB (the worker's own CUDA context ≈ 1.05 GiB counts as "used"), so
--gpu-memory-utilization ≥ 0.989 can never boot. The explicit pin skips profiling and the util
gate; 0.95 is a formality. Headroom was then proven by real 3×~501k soaks, not a profiler.VLLM_KVQ_TILES=6666 (compact KV rows) — the sparse-MLA layers' fp8_ds_mla KV row is 656 B/token:
a 512 B fp8 latent + a 128 B RoPE slot that is provably inert on this NoPE model
(qk_rope_head_dim=0). The patch drops the dead slot and stores the latent as four fp6 tiles →
416 B/row, 5.87 KiB/tok effective (measured). That is what lets a 548,560-token pool + the MTP
draft's KV live in a 3.07 GiB/rank pin — canonical rows would need ~4.4 GiB that this card does not
have. Rows are re-staged to canonical 656 B on read; the attention cubin itself is unmodified.--max-num-seqs 1 --max-num-batched-tokens 1024 — the sparse-MLA prefill carries an O(context)
transient (~4+ GiB at 500k). seqs 2 / batched 2048 passes short-context quality but OOMs at a real
503k prefill. Single-sequence is the 512k use case; concurrent requests queue.Rebuilt from the pristine parent by scripts/convert_fp8attn.py + scripts/convert_lmhead.py
(bit-for-bit reproducible — determinism experimentally verified). 425 tensors BF16 → F8_E4M3
weight + FP32 weight_scale_inv block scales, dispatched via a MIXED_PRECISION manifest in config.json:
| family | FP8 block | size | round-trip err |
|---|---|---|---|
| routed experts (incl. MTP draft) | NVFP4 — unchanged, LibertAIDAI's work | 163.3 GiB | — |
| KDA q/k/v + b/f_a/g_a (34 layers) | [32,32] | 6.46 → 3.23 GiB | ~2.3–2.4% |
| o_proj, all 46 layers | [128,128] | 3.63 → 1.81 GiB | ~2.3% |
| MLA q_a/kv_a/q_b (12 layers) | [128,128] | 0.75 → 0.38 GiB |
Kept BF16 on purpose: kv_b_proj (MLA absorbs it to BF16 at runtime anyway), KDA f_b/g_b
(the upstream authors are most explicit about delta-rule precision there), the sparse indexer,
embed_tokens, the entire vision tower, norms, routers, convs.
The MLA/MLP families measure near-lossless (0.12–0.16%) because base GLM-5.3 is natively block-FP8 — those BF16 tensors are dequantized copies. The KDA projections sit at the ~2.3% E4M3 floor: a real numerics change on a precision-sensitive path, which is why the quality gate exists. We do not call the attention conversion lossless; it passes a ≥95% teacher-forced gate with protocol, n, and raw dumps published.
Block [32,32] for the fused KDA in_proj: vLLM fuses q,k,v,b,f_a,g_a into one module with a 32-row b
shard at a non-128-aligned offset; the block must be square and divide 32.
Integrity: 121 shards, 150,651 tensors, index↔header cross-check zero mismatches; total_size exact
(186,674,168,184 B = 173.85 GiB).
All numbers measured on 2× RTX PRO 6000 Blackwell Workstation (96 GB, TP2, PCIe), vLLM
0.1.dev20051+g487ecf187 in the patched image. Raw dumps ship in scripts/.
Memory (per rank, vLLM's own ledger). At identical flags to the old offloaded config, FP8 attention+MLP moved the KV pool 340,163 → 645,997 tokens (+90%) and max loadable context 315,392 → 524,288. Offload-free at 524,288: weights+non-torch 84.77 GiB/rank (MTP-off −2.01, vision-skip −0.53, FP8 lm_head −0.29 — each landed to the decimal), KV pinned 3.95 GiB = 526,825 tokens at fp8 (~7.84 KiB/token/rank).
The fit ladder (how the flags were found):
| # | attempt | result |
|---|---|---|
| A | util 0.985, seqs 2, b2048, MTP/mm off | FAIL — avail 3.08 vs 3.92 GiB; O(maxlen) profile peak |
| B | util 0.99 + trims | FAIL — util ≥0.989 can never boot (own CUDA context in the reading) |
| B3 | --kv-cache-memory pin | boots + serves; pool 552,543 tok |
| C1 | + tuned kernels, seqs 2 | quality PASS, 101.7 tok/s — OOM at real 503k prefill (O(context) indexer transient) |
| C2 |
The everything lane at 524,288 (one boot, full gate battery): KV pool 548,560 tokens from a 3.07 GiB/rank pin (5.87 KiB/tok measured vs 5.99 predicted from the row math); MTP active at 95.8% acceptance; 144.5 tok/s median short-context decode; vision spot-set 10/10, re-probed clean after the long rounds; repetition probes at 251k clean (worst 4-gram 3); 3× fresh ~501k-token prefills in 90–96 s each, zero hard OOMs; min-free watermark 270 MiB. The fp6 rung is the only one that closes: the bit-identical 8888 rung would need ~0.66 GiB more pin than that margin can fund.
Quality gates (protocol: teacher-forced, identical context, n=271 scored positions; plus 10 greedy
generations; scored against the unmodified NVFP4 parent under the unpatched vendor image — reproduce with
scripts/compare.py):
| metric | MLP-only FP8 | + attention FP8 | pre-recal | shipped (recal scales) | everything lane (fp6 KV) |
|---|---|---|---|---|---|
| teacher-forced top-1 | 97.4% | 97.05% | 96.68% | 95.57% (gate ≥95) | 94.10%² |
| greedy agreement to 1st divergence | 94.7% | 90.5% | 60.0% (n=25, noisy)¹ | 90.82% | — |
| long-gen degeneracy |
¹ Greedy chains on a reasoning model diverge at the first flipped token; teacher-forced is the signal. ² Same weights; the delta is purely the fp6 KV-cache rung at run time (96.31% agreement vs the text-only lane's own dump). Disclosed, not gated away: use the text-only flags when the last 1.5 points matter more than MTP + vision + the 512k-with-them ceiling.
The patched image is bitwise neutral for BF16-attention checkpoints (100.00% agreement,
|Δlogprob| = 0.00000 vs the vendor image). MTP acceptance stays 81–98%. 17×23 → 391 on every
configuration, including served at the 524,288 ceiling.
The kernel story. vLLM ships zero SM120-tuned w8a8 block-FP8 triton configs. The fused KDA
in_proj (N=12576, K=4096, block [32,32]) ran a default config at 313 µs/call — 43.8% of decode GPU
time. One autotuned JSON (ships in serving/configs/, 22.3 µs at M=1) plus dropping CPU offload took
the checkpoint 66 → 142 tok/s at 114k. Same BLOCK_SIZE_K k-split, so numerics unchanged. The
block-128 GEMMs already ran vendored DeepGEMM SM120 kernels at ~1.35 TB/s and were left alone.
One honest negative. MLP-only FP8 (round 1) was a net loss: vLLM reserves the profiled activation peak, and activating the FP8 block-GEMM path costs a fixed +0.80 GiB/rank regardless of layer count — it outspent the 0.71 GiB saved (pool −2.8%). Only when attention (+2.81 GiB/rank) joined did the fixed cost amortize 4:1. Building a partial-FP8 derivative: go all-in or don't bother.
Input-scales provenance. The parent's expert input_scale tensors went through three revisions in
one day: an input_scale=1.0 placeholder (357b45cc), its retraction
(discussion #7 — small-amax
blocks underflow to zero, input-dependent repetition), and a same-day recalibration (caca4e6a).
This repo pins caca4e6a and gates it directly: 95.57% TF (gate ≥95), repetition probes clean at
225k and 259,632-token contexts (worst 4-gram repeat 3 and 2 vs fail ≥12), 503,374-token round in
80.4 s. Dumps: scripts/results-recal.json, results-longrep-recal.json, results-longrep-prod-topup.json.
serving/ patches against; and the vLLM project (Apache-2.0).8590bf9c) around the same time.markdown
├─ model-000{01..120}-of-00120.safetensors # 173.9 GiB: NVFP4 experts + block-FP8 + BF16├─ model-input-scales.safetensors # recalibrated scales (parent @caca4e6a)├─ config.json # MIXED_PRECISION manifest (38k+ entries)├─ serving/ # matched-pair runtime patch (Apache-2.0, vLLM-derived)│ ├─ Dockerfile ├─ kda.py ├─ model.py ├─ modelopt.py ├─ configs/*.json│ └─ everything-lane/ # + MTP/vision/512k patch set (compact KV, bounded indexer)└─ scripts/ # conversion + acceptance harness + raw quality dumps
serving/ = the public cstechdev/vllm:glm53-flash-nope-sm120-cu130-20260826-r1 image (the
chriswritescode-dev/glm-5.3-flash-sm120
overlay of the official per-model image) plus exactly four files: the KDA and MLA quant_config
passthroughs, the FP8_BLOCK dispatch + FP8 lm_head loader, and the SM120-tuned triton config.
serving/everything-lane/ layers the MTP + vision + 512k patch set on top: compact KV rows
(VLLM_KVQ_TILES), the bounded-memory sparse-MLA indexer, and the embedding/vision quant_config
passthroughs. Apache-2.0, SPDX headers retained; base image pulled from Docker Hub, not redistributed.
serving/ patched images.--limit-mm-per-prompt '{"image":1,...}'); vision ships
unquantized (byte-identical to parent). Video untested.--max-num-seqs 1); concurrency was not the target.Weights MIT © 2026 Z.AI Co., Ltd (LICENSE verbatim). Chain:
zai-org/GLM-5.3-Flash →
LibertAIDAI/GLM-5.3-Flash-NVFP4 (caca4e6a;
NVFP4 expert quantization and input scales are their work) → this repo (block-FP8 attention/MLP/lm_head).
serving/*.py Apache-2.0 (vLLM-derived). Conversion CPU-only, deterministic, calibration-free;
scripts/convert_fp8attn.py + convert_lmhead.py reproduce the checkpoint bit-for-bit.
Not affiliated with Z.ai / Zhipu or LibertAI.
--compilation-config '{"cudagraph_capture_sizes":[1,2]}'everything-lane/indexer.py| 0.12–0.16% |
| shared experts + dense MLP | [128,128] | 2.86 → 1.43 GiB | 0.16% |
| lm_head | [128,128] | 1.18 → 0.59 GiB | 2.64% |
| ACCEPTANCE PASS — 3×503k clean, watermark 290 MiB, zero failures |
| D | + MTP + vision at 524,288, stock KV rows | cannot fit — draft KV alone needs ~1.5 GiB the card doesn't have |
| D2 | + VLLM_KVQ_TILES=6666, pin 3.07 | ACCEPTANCE PASS — every gate on one boot (below) |
| none |
| none |
| none |
| none, incl. 2,712-tok gens at 225k & 259,632 ctx |
| none, incl. 2,912-tok gens at 251k ctx |