GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Bizuayeu
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
Bizuayeu
Model Tree
nvidia/GLM-5.3-Flash-NVFP4
Input Modalities
Output Modalities
Supported Functionality
| Source | nvidia/GLM-5.3-Flash-NVFP4, revision 423acf37583782c51c142d145aef733d72943d93, checksum-verified snapshot |
| Upstream | zai-org/GLM-5.3-Flash (MIT, Copyright (c) 2026 Z.AI Co., Ltd) |
| Repacked to W4A16 NVFP4 (group 16, no calibration data) | The KDA projections of the 34 linear-attention layers (q_proj, k_proj, v_proj, b_proj, f_a_proj, g_a_proj, f_b_proj, g_b_proj, o_proj); on the 11 MLA layers q_a_proj, kv_a_proj_with_mqa, q_b_proj, kv_b_proj, o_proj and the indexer wq_b; and lm_head |
| Left as shipped | Routed experts (NVIDIA's NVFP4), router, norms, embeddings, shared experts, the vision tower, the indexer wk / weights_proj, the MTP draft layer |
| Files | 18 safetensors shards (195 GB, 182 GiB); config.json and hf_quant_config.json carry the quantization declarations; tokenizer and chat template from the source; NVIDIA's model card kept as README.nvidia.md |
| Conversion | requant/requant.py of tenhkspark/glm53-flash-nvfp4-2node at commit 8ee63a676c6e8550f49b52aec7c5c05b56eb5553 (Apache-2.0), locally extended to include lm_head; deterministic; about 5.5 minutes on one GB10 (CPU only). Recorded in config.json under quantization_config.producer (requant_target: "l", in_proj_layout: "split-qkv-bfg") |
| Tested on | Two GB10 systems (MSI EdgeXpert), TP=2 over QSFP/RoCE, vLLM 385dce36 in the setup's 1.19.0 image, Marlin W4A16 kernels, MTP draft depth 3, one or two active sequences (two with 6 GiB of KV per rank), 256K context with image input. Also three GB10 systems at TP=3 on a QSFP ring (the setup's 1.24.0, 2026-10-01): 63.69 GiB per rank, decode 51.00 / 30.10 / 39.48 tok/s on counting / prose / code, and a 1,038,423-token prompt answered at max_model_len 1,048,576 |
| License | MIT (NVIDIA's model card terms and the upstream Z.AI license, both preserved) |
The pinned vLLM (385dce36) does not route these modules through its NVFP4 path on its own, and this checkpoint declares the KDA input projection as q_proj / k_proj / v_proj plus one merged in_proj_bfg_a (b, f_a, g_a) instead of vLLM's six-in-one in_proj_qkvbfg_a. Loading therefore needs the two source overlays that the setup repository ships in its overlays/ directory:
| Overlay | Replaces in vLLM 385dce36 | SHA-256 | Base SHA-256 (the file it replaces) |
|---|---|---|---|
kda-quant-split.py | vllm/models/glm5next/nvidia/kda.py | 27a532ced5011543eb17b1e538a61415fd51d8df8ff904245f04030f3985f30b | 99cfc24678c0a99f2c813c20e3dcc30c8bfac4f55ec4299737689d5d51f472cd |
mla-quant-split.py | vllm/models/glm5next/nvidia/model.py |
An overlay pair belongs to one revision of this checkpoint (see Revisions); a mismatched pair fails at load. The KDA overlay above is the setup's 1.24.0 version: it copies f_a and g_a into fresh storage before their Marlin projections, which TP=3 (22 heads per rank) needs. At TP=2 it gave the same decode-check hashes and teacher-forced log-probabilities, bit for bit, as the earlier 144a835f…, which TP=2 profiles may keep. Plain vllm serve on this folder without the overlays does not start.
Above about 250K tokens this checkpoint also needs the slot-mapping guard (patch_slot_mapping, vLLM issue #53982) that the setup's images carry from 1.7.0: without it, the pinned vLLM's slot-mapping kernel reads past the end of a block-table row, and with this checkpoint's memory layout that faulted on both ranks between 248,954 and 252,958 tokens (details). On a runtime without the guard, keep requests at or below 245,000 tokens.
runtime.derived_checkpoint in the server TOML — path to this folder, requant_target = "l", and the two overlays with their SHA-256 and base SHA-256 — and start through its launcher. server preflight refuses the launch unless each overlay has its declared hash and marker and the image's own target file has the declared base hash, so an overlay made for another image cannot be mounted. examples/server.axl.example.toml carries all of it, with prefix-page dedup and two active sequences; replace its paths, image IDs and node details, and set each host's optional cpuset_cpus to performance cores from that host's own topology (CPU placement). Details: server configuration.Two GB10, TP=2, MTP depth 3, temperature 0, the setup's 1.19.0 image. Both columns were measured on 2026-09-28 in one window: the pair switched from this checkpoint's serving profile to the source checkpoint, then to this checkpoint at one sequence (for repeatability), then back. This checkpoint is the profile the setup's reference pair serves (the AXL example: prefix-page dedup, two active sequences, 6 GiB of KV per rank), measured with one request in flight unless the row says otherwise. The source checkpoint is the setup's distributed defaults (one sequence, 3 GiB of KV per rank). Both ranks' workers ran on performance cores (CPUs 5–9 and 15–19); with either rank on efficiency cores decode falls to about a third. All nodes ran with the GPU clock capped at 2,200 MHz, and each stage waited for temperatures to come down first; against this checkpoint's profile without the cap (2026-09-27), prefill was 1.8% slower and long inputs 0.8–1.9% slower. Medians of three runs; ranges and conditions are in the setup repository's benchmarks.
| Category | Measure | Source checkpoint | This checkpoint |
|---|---|---|---|
| Prefill | 38,962-token prompt (tok/s) | 1,233.4 | 1,259.1 |
| Decode | After a 2,048-token prompt, counting / prose / code (tok/s) | 32.59 / 21.12 / 28.21 | 46.73 / 30.06 / 39.48 |
| Decode | 512 tokens after a short prompt (tok/s) | 27.18 | 42.63 |
| Decode | sparkDash DecodeBench, 128 tokens, structured / prose / code / json (tok/s) |
The sparkDash code prompt changed from 108 to 66 tokens on 2026-09-26, so its code column cannot be compared with this card's earlier revision.
Quality, teacher-forced NLL on the setup's four fixed corpora (same inputs; the serving profile, two sequences, deterministic Inductor configs; the same values on the setup's 1.12, 1.13 and 1.19 images):
| Corpus | Source checkpoint | This checkpoint | Change |
|---|---|---|---|
| Japanese | 1.5963 | 1.6270 | +1.9% |
| English | 2.0241 | 1.9946 | −1.5% |
| Code | 0.9479 | 0.9601 | +1.3% |
| Mathematics | 0.5931 | 0.6275 | +5.8% |
The earlier figures (2026-09-21, one-sequence profile before the setup's 1.12 launch-state fix) were 1.6645 / 2.0024 / 1.0031 / 0.6279.
Since the setup's 1.12.0 (deterministic Inductor configs on both ranks), every launch checked has computed in the same numerical state; before it, relaunches fell into one of three states that differed in how a near tie falls. The setup's 1.19.0 ports vLLM #58454 (the kpool tail ring), which can change the compressed keys of pools built during decode past the indexer's index_topk (2,048): reference completions recorded on earlier images are not a baseline for a context past that length. The setup checks every launch after a switch (weight digest, decode check, kernel hashes on both ranks); see its validation. Decode speed under MTP depends on how predictable the text is (counting is about 1.5 times as fast as prose); the draft is the checkpoint's own MTP layer, no extra weights.
| Date | Change | Overlay pair |
|---|---|---|
| 2026-09-22 (current) | config.json / hf_quant_config.json declare the KDA input projection split into q_proj / k_proj / v_proj + in_proj_bfg_a (producer.in_proj_layout = "split-qkv-bfg"). Weights unchanged. Removes the prefill cost of the fused layout (one wide W4A16 Marlin GEMM), so prefill is now faster than the source checkpoint | kda-quant-split.py 27a532ce… (from the setup's 1.24.0; 144a835f… before, identical output at TP=2), mla-quant-split.py |
The weights and declarations have not changed since 2026-09-22; later commits edit this card only.
The weights are redistributed under the MIT terms stated by the NVIDIA model card and by the upstream Z.AI license; both notices are preserved (README.nvidia.md, and the Z.AI license text in the setup repository's LICENSES/). This repackaging adds no training data and no new terms. The conversion tool is Apache-2.0 and its license applies to the tool, not to these weights. No warranty; see the source model cards for intended use and limitations.
a9b1ea5ce05983973bc79954147809a17e0e5aa0f96eae47c6011306f51b533d48f60a65afab1afd737b282d109c6c11f97d881d3ec774442e174fab09c2f525 |
385dce36vllm serveconfig.json| 33.24 / 25.45 / 27.53 / 26.32 |
| 48.04 / 31.28 / 37.34 / 34.72 |
| Startup | Main-model weight loading, rank 0 (Loading weights took, s), with the setup's clone loading from its 1.19.0 | 122.6 | 120.7 (532.0 on the setup's 1.18.0, before the clone) |
| Long input | ~200K tokens, one passphrase at the midpoint (s, correct) | 178.9 and 176.6 (199,652 tokens) | 170.1 (199,649 tokens) |
| Long input | 255,950 tokens, one passphrase at the midpoint (s, correct) | 229.5 | 220.9 |
| Long input | 261,573 tokens, three-position reference, fenced prompt (s, correct 3 of 3) | 239.8 | 235.0 |
| Long input | Maximum capacity, 262,080 input + 64 output tokens (s, finite logprobs) | 253.0 and 252.7 | 244.8 and 244.7 |
| Concurrency | Two ~200K passphrase requests together | not served (one sequence; requests queue) | 226.1 s and 336.0 s, both correct, no preemption |
| Concurrency | Decode while another request runs, counting / prose (tok/s) | not served | 32.87 / 22.43 |
| Quality | tool-eval-bench, 69 standard scenarios | 91/100 | 88/100; the same three failures (TC-21, 43, 61), so its Safety Gate is not passed on either |
| Quality | FreedomBench, the pinned English 60 / the long-prefix pilot | 60 correct, no refusals / 6 of 6 | 60 correct, no refusals / 6 of 6 |
| Repeatability | Identical requests at temperature 0 | same completion nine times of nine within a launch; with max_num_seqs = 1, two sent together queue and each repeats its lone completion (18 of 18) | the same when served with max_num_seqs = 1 (18 of 18), and the decode check's completions were the same on all eight 1.19.0 launches. With max_num_seqs of 2 or more, a request that shares steps with another can get a different completion (measured on these weights and on the source checkpoint): the MoE kernel splits its sum along K by the number of expert blocks in the step, and a step shared with a prefill runs prefill-sized kernels |
| Memory | Weights loaded per rank (2026-09-22) / lowest available memory on the head during the bench | 95.76 GiB / 6.2 GiB (3 GiB of KV) | 91.38 GiB / 7.56 GiB (6 GiB of KV); 7.34 GiB with two 200K requests in flight; 10.50 GiB at 3 GiB of KV (2026-09-22) |
a9b1ea5c…| 2026-09-21 | Initial release; declarations for vLLM's fused in_proj_qkvbfg_a; prefill 1 to 3% slower than the source checkpoint | kda-quant.py 2e491a38…, mla-quant.py 88916cc1… |