canada-quant

glm-5.3-w4a16-mtp

🏆 Headline — where W4A16 wins (all measured, same-harness comparisons)

−70% weights: 177.7 GiB vs ~599 GiB BF16 → 4 datacenter GPUs instead of 8, or a 2× DGX Spark desktop pair at full 1M context.
vs NVFP4 (LibertAI, strongest community quant) — 2× DGX Spark: W4A16 serves 1,048,576 tokens there (KV pool 1.30×); the NVFP4 checkpoint OOM'd 9/9 boots at any context on identical nodes (dmesg-verified). Throughput @262K: +53% seq1 / +101% seq6.
vs NVFP4 — H100 TP=4, matched MTP: wins c1 +4.8% · c8 +13.6% · saturated c32 +14.0% (vs NVFP4's best c32 config); mtp-off fixed-drafter c32 +22.4% — the batch edge is the marlin INT4 MoE GEMM, not the drafter.
vs EXL3 (MiaAI-Lab 2×-Spark recipe) — independent matched-protocol head-to-head, FINAL: W4A16 leads single-stream +10.4% (33.0 vs 29.9 tok/s) and long-prefill +31–39% (~1.2K vs 864–914 tok/s @300K-class); KV capacity 1.30× vs 1.20×. The vendor README overstated decode ~2.2× and prefill ~1.2× vs independent measurement.
Quality: AIME 2026 85.0% (102/120, n=120 final, max thinking) — beats the matched EXL3 arm 96/120 (80.0%) by +6 correct (+5.0 pt), FINAL 2026-09-06; AIME 2025 parity with NVFP4 on H100 (0.8833 vs 0.9000, 0.42σ); GSM8K 0.97; GPQA-Diamond within noise.
SM120 (Blackwell): throughput parity with NVFP4 (±0.7%) with MTP paying at every concurrency — +69.9% c1 / +37.6% c8 / +47.8% c32.
4× H200 (SM90, 141 GB) — measured 2026-09-08 on AWS p5en.48xlarge, published recipe as-is (TP=4, MTP N=2): c1 200 · c8 672 · c32 1,234 · c64 1,507 · c128 1,751 · c256 ~1,970 output tok/s (8192/1024, thinking on), KV pool 6.19M tokens ≈ 23 concurrent 262K requests. Beats the best H100 row at every concurrency (+9% c1 / +10% c8 / +6% c32) and keeps scaling to c256. Prefix caching ON measured no gain on H200 (−2…−5%). 8× H200 TP=8: 210 · 823 · 1,636 · 2,063 · 2,385 · 2,674 (c1…c256), TPOT 4.5 ms single-stream. 2× H200 TP=2: 262K context with a 9.8× KV pool (c32 814 tok/s) and 1M context serving-validated (929K-token prompt admitted and decoded on two GPUs).
Long context: single-stream decode is KV-length-insensitive to ≥486K (up to 357 tok/s, bf16 KV); long-prefill ceiling ~6.1–8.5K tok/s aggregate across the full 3-KV × c1–c32 matrix.
Research-stack bug found and fixed: a 21-leg investigation root-caused a hang-forever admission wedge in the DFlash2 spec-decode stack and validated a fix to 256,388-token prompts (24/24 quality spot-checks pass).

Honest ledger: EXL3 leads mid-concurrency (c2 +67% / c4 +89%; ≈+81% cap-corrected c6); both sides saturate at c32 (EXL3 881–898 agg tok/s). SM120 vs NVFP4 is parity. The matched AIME-2026 quality arm vs EXL3 is FINAL 2026-09-06: W4A16 102/120 (85.0%) vs EXL3 96/120 (80.0%) — W4A16 wins. Results as of 2026-09-06; in-flight arms are labeled.

Why this quant exists

BF16 GLM-5.3-Flash needs eight 80 GB GPUs per replica; this artifact cuts weights to 177.7 GiB — four GPUs, or a dual DGX Spark desktop at 1M context.
Only the 36,288 routed-expert GEMMs are INT4 (symmetric, group-128, GPTQ); attention, router, shared experts, embeddings, the BF16 vision tower, and the BF16 MTP draft head (+40–50% single-stream) stay unquantized; canonical FP32 tensors preserved verbatim.
Not a new model — all capability comes from the base model.

Hardware

The model's context is 1M tokens (1,048,576). Each row is the largest context serving-validated on that config — x86 rows are KV-memory-bound, not model-bound.

Table with columns: Config, GPUs, Validated context, Serving stack
Config	GPUs	Validated context	Serving stack
DGX Spark GB10 (SM121), TP=2 + EP	2× 128 GB UMA	1M — KV pool 1,360,420 tok (1.30×)	DFlash2 drafter, fp8 KV — recipe below
RTX PRO 6000 (SM120), TP=4 + EP	4× 96 GB	512K (engine cap 524,288; 486K prompts measured)	MTP N=2, fp8 KV
H100 (SM90), TP=4 + EP	4× 80 GB	262K (256K prompts measured)	MTP N=2 or DFlash2, bf16 KV
H200 (SM90), TP=4 + EP	4× 141 GB

Quantization ran on 8× NVIDIA B300 (SM103) — see Quantization below; it is not a serving config.

Benchmarks vs competitors

Identical flags/prompts/hardware per row. Throughput = output tok/s, thinking ON.

vs EXL3 (MiaAI-Lab) — 2× DGX Spark, matched protocol, FINAL (2026-09-03)

EXL3 was brought to full serving at 1M context on the same hardware (KV pool 1,200,348 = 1.20×; W4A16 1,360,420 = 1.30×).
Like-for-like decode (TTFT excluded, 8K prompts, 256-token completions): W4A16 +10.4% c1 (33.0 vs 29.9 agg tok/s).
EXL3 leads mid-concurrency: c2 +67.4% (59.6 vs 35.6), c4 +89.1% (112.7 vs 59.6), c6 ≈+80.9% after correcting its max-num-seqs=4 scheduler cap; it sustains ~29 tok/s per stream at every concurrency (W4A16 steps down 33.0→11.2 seq1→6 on the 1M serve; the 262K serve holds ~20 at seq4+).
Saturated c32 (clean refire): EXL3 881–898 agg tok/s (32/32 streams, per-stream 27.4–28.1).
W4A16 +31–39% long-prefill (~1.2K tok/s vs 864–914 @300K-class prompts).
Vendor README overstated measured decode ~2.2× (claimed 62.9/51.7/37.1 per-stream c1/2/4 vs measured 26.6/21.2/17.0) and prefill ~1.2×.
Two mid-bench hardware deaths on the competitor node were quarantined and never banked; the c32 set was re-run clean.
Matched AIME-2026 quality arm: FINAL 2026-09-06 — W4A16 102/120 (85.0%) beats EXL3 96/120 (80.0%), +6 correct (+5.0 pt). EXL3 epochs 25/25/24/22, zero harness errors; 23/120 EXL3 rows hit the 131,072-token cap (W4A16: 16/18 misses were budget-exhaustion) — scored as written under the identical temp-0.0 / 131,072-token / n=120 config as our banked 85.0% arm.

4× H200 (SM90) — TP=4, published recipe, isl/osl 8192/1024, thinking ON (measured 2026-09-08, AWS p5en.48xlarge, image `glm53-flash-x86_64-cu130`)

Output tok/s, warm (second grid after boot — the first grid after boot under-reads by up to 2.2× at c8 on H200; cold rows are kept in the raws, never banked). Client: vllm bench serve --random-range-ratio 0.2 --ignore-eos, temp 1.0 / top_p 0.95, n = 8/32/64/256/512/768.

Table with columns: config, c1, c8, c32, c64, c128, c256, MTP accept, TPOT p50 ms (c1 → c256)
config	c1	c8	c32	c64	c128	c256	MTP accept	TPOT p50 ms (c1 → c256)
MTP N=2, prefix caching OFF (published) — FINAL, alone, mean of 3 reps	195	698	1,258	1,529	1,789	1,954	53–54%	4.5–4.9 / 10 / 20 / 38 / 66 / 118 — GSM8K-200 on this recipe on H200: 0.99

† measured with a second identical TP=4 serve benching concurrently on the other 4 GPUs (interference vs alone ≤ 5%, calibrated on the same recipe). 8-GPU guidance (measured): two TP=4 replicas behind a load balancer deliver 3,067 / 3,550 / 3,911 aggregate tok/s at total c128 / c256 / c512 versus 2,399 / 2,681 / 2,834 for one TP=8 endpoint (+28…38%), with zero cross-replica interference; TP=8 wins single-stream (217 vs 196 tok/s, TPOT 4.2 ms) and holds one 7.8M-token pool for very long requests. DP=2×TP=4 was measured and is not recommended. DFlash2 runs on H200 only on the upstream nightly with two small patches (maintained by the authors) and does not beat MTP N=2 on the pinned image. H100 (louse, same recipe, warm): c1 180 / c8 254 / c32 804; best H100 row ever (PC ON): 184 / 610 / 1,161 — H200 with PC OFF beats it at every point. Raws + scripts maintained by the authors; available on request.

vs NVFP4 — H100/H200 (SM90), TP=4, MTP both sides, isl/osl 8192/1024

Table with columns: concurrency, W4A16 MTP N=2 (this), NVFP4 MTP N=2, NVFP4 MTP N=5 (its best c32 config)
concurrency	W4A16 MTP N=2 (this)	NVFP4 MTP N=2	NVFP4 MTP N=5 (its best c32 config)
1	183.89	180.96	175.44
8	249.94	251.14	220.05
32	1161.13 †	770.66	1018.13
MTP acceptance	52–55%	~52%

Headline: +4.8% c1 / +13.6% c8 / +14.0% saturated c32 vs NVFP4's banked N=5 config.
† c=32 with --gpu-memory-utilization 0.92 (KV-pool capacity): 1161.13 tok/s (independent tuned run: 1145.05). The c=32 lever is KV-pool size, not a kernel gap.
Fixed-drafter 2×2 re-confirm (both arms on the corrected #54374-port overlay): matched c1/c8 within noise (180.22/254.39 vs 182.13/252.89); saturated c32 win holds (+18.1%, 804.02 vs 680.99).
MTP-off on both: W4A16 leads +2.7% / +10.5% / +22.4% at c1/c8/c32 — the c32 advantage is the marlin INT4 MoE GEMM at batch, not an MTP artifact.
MTP adds +41.1%/+14.3%/−4.0% (W4A16) and +46.4%/+25.6%/−0.4% (NVFP4) at c1/c8/c32 on SM90.

vs NVFP4 — RTX PRO 6000 (SM120, Blackwell), TP=4, matched MTP N=2

Table with columns: concurrency, W4A16 (this), NVFP4, Δ
concurrency	W4A16 (this)	NVFP4	Δ
1	109.90	109.34	+0.5%
8	318.54	319.46	−0.3%
32	534.37	530.75	+0.7%

Parity within noise (±0.7%); verbatim-methodology re-grid: c1 106.97 (−2.7%, 8-prompt high-variance class), c8 320.51 (+0.6%), c32 539.63 (+1.0%), MTP acceptance 50–53% — no stack drift.
MTP pays at every concurrency on SM120: +69.9% c1 (106.97 vs 62.96), +37.6% c8 (320.51 vs 232.89), +47.8% c32 (539.63 vs 365.02) — a sign flip vs SM90 (drafter costs ~4% at saturated c32 there). Keep MTP on at every concurrency on SM120.
KV-dtype at batch: fp8 vs bf16 costs nothing at c1/c8 (±0.05%) and ≈4.7% at c32 (365.02 vs 382.07) — lever ranking MTP ≫ KV dtype. (auto is the working bf16-KV path; bf16/bfloat16 variants fail on this stack.)

Long-context decode at batch (SM120, TP=4, 524K ctx, MTP N=2 — decode-isolated, TTFT excluded)

Per-stream decode tok/s (full-contention cohort bands — serialized prefills split stages into early/late cohorts, so bands, not blended medians):

Table with columns: KV length, KV dtype, c1, c4, c8, c32
KV length	KV dtype	c1	c4	c8	c32
8K (aggregate grid)	fp8	106.97	—	40.06	16.86
138.6K	fp8	143.94	2.92–7.53	2.49–3.74	2.24–2.72
138.6K	bf16 (`auto`)

* n=1 short completion, noisy. † 21/32 rows — 11 bench-client TimeoutErrors disclosed in the authors' bench records. ‡ the fp8-KV c1 row recorded an absurd >480K tok/s (bench artifact) and is never cited. Additional bands: 277K c2 5.03, c6 2.29–3.38; 486K c6 2.34–3.68 (bf16) / 2.34–3.60 (fp8); 138.6K fp8 c2 9.58.

Single-stream decode is KV-length-insensitive to ≥486K (143.94 → 177.15 → 357.34 tok/s at 138.6K/277K/486K bf16 — rises, if anything).
At-batch collapse saturates with KV length: ~13× per-stream drop at c8 vs the 8K grid, then flat ~2.2–3.7 tok/s/stream at c4–c32 from 138.6K through 486K — mechanism is target-model forward compute at batch × long-KV (~4.4M tokens of live KV at c32), not the drafter.
KV dtype is a c1-only ~5% lever (bf16 +5.7% @138.6K); at-batch bands overlap fp8/bf16 — the long-KV-amplifies-fp8-cost hypothesis is refuted.
MTP acceptance never collapses: c1 ≈80–86% → c8 73.9% (n=1,286) → c32 71.6% (n=4,714) at 138.6K; 60.0% single-stream at 486K (small sample, disclosed). The saturated c32 stage decoded 2.53 tok/s/stream while the drafter accepted 71.6% — conclusively not acceptance-driven.
At-batch lever ranking FINAL: MTP drafter ≫ everything measured; KV dtype ≈ c1-only ~5%; drafter acceptance is not the bottleneck.

Long-prefill at batch (SM120) — full 3-KV × c1–c32 matrix COMPLETE

Single-stream prefill: ~6,100–7,100 tok/s for 277K–486K prompts (TTFT first request 39.2–79.3 s).
At batch, long prefills are essentially serialized (stage wall ≈ N × single-stream time).
Aggregate prefill ceiling ~6.1–8.5K tok/s — KV-length-insensitive AND batch-insensitive:
138.6K: 6,735 (c1) → 7,134 (c8) → 7,897 (c16) → 8,486 (c32) tok/s.
277K: 5,211 (c1) → 7,177 (c8) → 7,784 (c16) → 7,357 (c32) tok/s.
486K: 6,136 (c1) → 6,892 (c8) → 7,483 (c16) → 6,988.6 (c32) tok/s.
Serialization ratios (wall / N×c1): 138.6K 0.94/0.85/0.79 · 277K 0.73/0.67/0.71 · 486K 0.89/0.82/0.88 (c8/c16/c32) — deeper-KV rows behave closer to strict serialization.
mnb lever fully mapped: 8192 ✓ (validated production config) → 16384 ✓ (validated fallback, ceiling within ~1% at both 277K and 486K) → 32768 ✗ (CUDA OOM at 486K — 3.98 GiB needed vs 1.70 GiB free at mem-util 0.95). Ceiling is chunk-size-insensitive → attention/forward compute-bound, not scheduling-bound.
Operational guidance: admission pacing / prefill scheduling is the only lever that moves long-context batch UX (mnb tuning and MTP do not address prefill serialization).

2× DGX Spark GB10 (SM121), TP=2, DFlash2 block-diffusion spec-decode

8K prompts, 256-token completions, aggregate tok/s:

Table with columns: config, seq1, seq2, seq4, seq6
config	seq1	seq2	seq4	seq6
W4A16+DFlash2 @262K (this)	33.29	42.01	80.98	121.81
NVFP4-MTP4 reference	21.8	41	66	60.5
Δ	+53%	+2.4%	+23%

256K-token single prompt on the 1M serve: 30.85 tok/s decode (ttft 5.8 s).
Structural: the NVFP4 checkpoint (90.67 GiB/node weights vs ~30 for W4A16) OOM'd all 9 boot attempts across 1M/300K/262K on the 121 GiB unified-memory nodes (dmesg-verified) — it cannot serve at any usable context on 2× Spark.

Quality

AIME 2026 (30 problems, max thinking, 131,072-token budget, DFlash2 stack, 2× Spark): 102/120 = 85.0% — n=120 FINAL. Epochs 28/30, 25/30, 26/30, 23/30; 16/18 misses are budget-exhaustion truncations, 2/18 genuine wrong answers. The LibertAI NVFP4 external AIME-2025 reference (0.9000) remains +5.0 points ahead; the matched AIME-2026 EXL3 arm is FINAL 2026-09-06: 96/120 (80.0%) — W4A16 wins by +6 correct (+5.0 pt).
AIME 2025 n=120, 4×30, max thinking, 131,072 budget — H100 (SM90): parity — W4A16 0.8833 (106/120) vs NVFP4 0.9000 (0.42σ). An earlier 0.7833 read was a serving-stack drafter-corruption bug (upstream vLLM #54374), since fixed.
AIME 2025 — RTX PRO 6000 (SM120): raw-arm deficit real (~2σ; 0.8083/0.8000 pooled vs 0.9000), mechanism decomposed and fix VALIDATED. ≈63% of the deficit is a mechanical budget-wall (EMPTY-answer rate 11.7–14.2% vs NVFP4's 3.3%; doubling the budget recovered +5.8 pt); ≈37% is SM120 kernel numerics (near-tie argmax flips vs SM90, logit-quantified).
Fix stack VALIDATED (zero GPU cost): whole-trace commit hook + thinking-budget force-commit (98,304-token cap) lifts the score to 0.8833 two-way / 0.8917 hook-credited — within noise of NVFP4 0.9000 (0.42σ / 0.21σ).
Production-overlay pilot passed (committed 0.8917, zero modified non-empty responses across 120 requests); production adoption remains gated on the eval sign-off chain (owner decision pending).
GSM8K: parity — 0.9704–0.9750 across all three quants on both x86 rigs.
GPQA-Diamond n=198 @131k: 0.8586 vs 0.8687 NVFP4 (SM90), 0.8586 vs 0.8737 (SM120) — within noise.
Matched AIME-2026 single-epoch arms on x86 (n=30, max thinking): W4A16 24/30 (RTX PRO 6000) and 22/30 (H100) vs NVFP4 23/30 — within noise at n=30.

Serving

The same checkpoint serves on every NVIDIA architecture from Hopper through Blackwell. Each recipe is the exact config the benchmarks ran.

SM90 — H100 / H200 (recommended: TP=4)

Re-download config.json if your copy predates 2026-09-08 (commit ece4697): older copies fail in vLLM with KeyError: 'layers.0.mlp.gate_up_proj.weight' — see the vLLM serving note under Details. Weights are unchanged.

Image vllm/vllm-openai:glm53-flash-x86_64-cu130 (CUDA 13.0). Gotcha: always pass --max-num-seqs ≤512 — the default 1024 exceeds this hybrid linear-attention model's 512 Mamba/KDA-state cache blocks.

bash
docker run --gpus '"device=0,1,2,3"' --ipc=host --network=host --rm \
  -v /models:/models vllm/vllm-openai:glm53-flash-x86_64-cu130 \
  vllm serve /models/glm53-flash-w4a16-mtp \
    --served-model-name glm53-w4 \
    --tensor-parallel-size 4 --enable-expert-parallel \
    --max-model-len 262144 --max-num-seqs 512 \
    --block-size 64 --gpu-memory-utilization 0.92 --no-enable-prefix-caching \
    --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
    --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
    --trust-remote-code --port 8000

MTP num_speculative_tokens: 2 is the sweet spot on Hopper (52–55% acceptance, best c1/c8); N=5 collapses acceptance (52→30%). H200 N=1 / N=3 / MTP-off and TP=8 rows are in flight (2026-09-08).
H200 (141 GB) with the recipe above, unchanged — engaged: Using 'MARLIN' WNA16 MoE backend, FLASH_ATTN_MLA_SPARSE, SpeculativeConfig(method='mtp', num_spec_tokens=2), attention block auto-set to 1152 tokens (mamba page alignment; --block-size 64 is inert), GPU KV cache size: 6,194,528 tokens (76.5 GiB free after the 45.25 GiB/GPU weights). Prefix caching ON measured −2…−5% vs OFF at every concurrency → keep --no-enable-prefix-caching. Optional for c≳170 with N=2: --max-cudagraph-capture-size 1536 keeps large decode batches inside full CUDA graphs (default cap 512 tokens/step; costs ~0.4M tokens of KV). Same recipe also boots on upstream vllm/vllm-openai:nightly-x86_64 (v0.28.1rc1.dev516+, includes PR #53906): FLASHINFER_MLA_SPARSE_SM90 backend, ~6.06M-token pool, −22% c1 / −6% c32 / tie at c64–c128 vs the pinned image (and that backend IMAs on ≥131K prompts — pass --attention-backend FLASH_ATTN_MLA_SPARSE on the nightly).

SM120 — RTX PRO 6000 (Blackwell)

Image cstechdev/vllm:glm53-flash-nope-sm120-cu130-20260826-r1 (SM120-patched build).

bash
docker run --gpus '"device=4,5,6,7"' --ipc=host --network=host --rm \
  -v /models:/models cstechdev/vllm:glm53-flash-nope-sm120-cu130-20260826-r1 \
  vllm serve /models/glm53-flash-w4a16-mtp \
    --served-model-name glm53-w4 \
    --tensor-parallel-size 4 --enable-expert-parallel \
    --max-model-len 262144 --max-num-seqs 64 --max-num-batched-tokens 8192 \
    --kv-cache-dtype fp8 --gpu-memory-utilization 0.92 --enable-prefix-caching \
    --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
    --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
    --trust-remote-code --port 8001

KV fp8 is required at 262K on 96 GB cards; --max-num-seqs 64 is bench-validated (48/36 fallback ladder); MTP N=2 matches NVFP4 to ±0.7%.

SM121 — 2× DGX Spark GB10 (desktop, 1M context)

Full 1,048,576-token context on two Sparks over RoCE (TP=2, expert-parallel) with the DFlash2 block-diffusion drafter. One launcher on both nodes; worker rank first, wait 25 s, then head rank (mandatory stagger).

Current production config (live since 2026-09-08): CUDA graphs ON + GMU 0.90. The published @1M numbers (decode 32.99/35.61/59.61/67.05, AIME26 102/120) were measured on the previous banked config (GMU=0.795, --enforce-eager); the current config re-measured equivalent within run-to-run noise (graphs-on vs eager decode wash), and the GMU raise itself was pre-validated (@262K staging: c2 +38.7%, c6 +10.6% vs 0.795).

Production launch (current):

bash
# rank1 (worker) FIRST → wait 25 s → rank0 (head):
MAX_MODEL_LEN=1048576 KV_CACHE_MEM=9663676416 GMU=0.90 EAGER=0 GRAPHS=1 bash launch_dflash2_tp2.sh <rank>

The full launcher is inlined below — the card is self-contained (the script previously lived out-of-tree). Launcher defaults = production for the remaining knobs: MAX_NUM_SEQS=6 BLOCK_SIZE=2304 SPEC_NUM_TOKENS=7.

Engine args the launcher produces (weights bind-mounted at /models/glm-5.3-w4a16-mtp; set MODEL_DIR to your local weights path):

bash
vllm serve /models/glm-5.3-w4a16-mtp/ \
  --served-model-name glm-5.3-flash --host 0.0.0.0 --port 8000 --trust-remote-code \
  --tensor-parallel-size 2 --enable-expert-parallel \
  --gpu-memory-utilization 0.90 \
  --max-model-len 1048576 --max-num-seqs 6 --block-size 2304 \
  --speculative-config '{"method":"dflash","model":"/models/dflash2-draft","num_speculative_tokens":7}' \
  --kv-cache-dtype fp8_e4m3 --kv-cache-memory 9663676416 \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[1,2,4,8,16,24,32]}' \
  --tool-call-parser glm47 --enable-auto-tool-choice --reasoning-parser glm45 \
  --default-chat-template-kwargs '{"enable_thinking":true}' \
  --distributed-executor-backend mp --nnodes 2 --node-rank <0|1> \
  --master-addr <HEAD_IP> --master-port 29521    # + --headless on rank1

Stack + topology:

Image note (2026-09-11): the pinned SM121 image radixark/vllm-glm53-flash:sm121-v11-dflash2 is no longer published — its Docker Hub repo returns 404 as of 2026-09-11. Our fleet runs a locally cached copy (an aarch64/CUDA-13.0 SM121 vLLM build carrying the DFlash2 serving patches; the stock upstream DFlash2 image cannot boot this drafter — see the defect section below). Until an upstream stack lands, the reproducible paths are: (a) upstream vLLM main (≥ vllm-project/vllm#53906) serves these weights on Spark with MTP — without DFlash2; (b) the exact patched SM121 build recipe is maintained by the authors and is available to collaborators on request.
Drafter incoai/GLM-5.3-Flash-DFlash2 mounted at /models/dflash2-draft; weights = this repo (178 GiB, bind-mounted into the container; config.json must be the current published one (the ignore list carries both HF and vLLM module spellings; older 765-entry copies fail to load)).
Drafter internals: block 8, selector_rank 256, top_k 16, layers [5,14,24,33,42]; 7 speculative tokens (any other count boot-wedges).
KV pool: 1,360,420 tokens fp8_e4m3 = 1.30× a full 1M request.
RoCE fabric: NCCL_IB_HCA=rocep1s0f1, GID index 3, RoCEv2, 192.168.102.0/24 (enp1s0f1np1), — path at 97.98 Gbps line-rate (direct-attach measured worse).

Full launcher (verbatim, sha256 440ca004b719…):

bash
#!/usr/bin/env bash
# GLM-5.3-Flash W4A16 + DFlash2 speculative decoding — TP2 across two DGX Spark GB10 (SM121) nodes.
# Adapted from tonyd2wild/GLM-5.3-Flash-NVFP4-DFlash2-2x-DGX-Spark/launch-glm53-vllm-tp2-dflash2.sh
# (their 46.9 tok/s config) for a two-Spark RoCE fabric:
#   - W4A16 weights (/home/pcozz/models/glm-5.3-w4a16-mtp/) instead of NVFP4
#   - fabric: rocep1s0f1 / enp1s0f1np1 / 192.168.102.0/24
#   - proven W4A16 serve args (EP on, fp8_e4m3 KV, block 2304, seqs 6)
#   - wedge-safety: graceful docker stop -t 30 (NEVER rm -f a GPU-active container), --memory=118g cap
# Usage: launch_dflash2_tp2.sh <0|1>  — worker (1) FIRST, wait 25s, then head (0).
# Env knobs (defaults = 262K serve):
#   MAX_MODEL_LEN  (default 262144; 1048576 = architecture max, needs KV_CACHE_MEM raise)
#   KV_CACHE_MEM   (default 3221225472 = 3 GiB -> 310,292 tok pool, ~10.38 KB/tok fp8;
#                   1M pool needs ~10.9 GiB -> use 11811160064 = 11 GiB -> ~1.09M tok;
#                   safer 1M value: 9663676416 = 9 GiB -> ~1.36M)
#   MODEL_DIR      (default /home/pcozz/models/glm-5.3-w4a16-mtp = W4A16)
set -euo pipefail
MAX_MODEL_LEN="${MAX_MODEL_LEN:-262144}"
KV_CACHE_MEM="${KV_CACHE_MEM:-3221225472}"
MAX_NUM_SEQS="${MAX_NUM_SEQS:-6}"
MODEL_DIR="${MODEL_DIR:-/home/pcozz/models/glm-5.3-w4a16-mtp}"
# DFlash2 drafter lever: override num_speculative_tokens (default 7; any other count boot-wedges).
SPEC_NUM_TOKENS="${SPEC_NUM_TOKENS:-7}"
# EAGER=1 (default) = --enforce-eager; current production passes EAGER=0 GRAPHS=1 (CUDA graphs ON).
EAGER="${EAGER:-1}"
if [ "$EAGER" = "1" ]; then EAGER_FLAG="--enforce-eager"; else EAGER_FLAG=""; fi
# Note: engine may raise block-size to >= mamba page size (see boot log).
BLOCK_SIZE="${BLOCK_SIZE:-2304}"
# Banked default 0.795; current production passes GMU=0.90.
GMU="${GMU:-0.795}"
# GRAPHS=1 + EAGER=0 = FULL_AND_PIECEWISE capture sizes 1-32 (current production).
GRAPH_ARGS=()
if [ "${GRAPHS:-0}" = "1" ]; then GRAPH_ARGS=(--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[1,2,4,8,16,24,32]}'); fi

NODE_RANK="${1:?usage: launch_dflash2_tp2.sh <0|1>}"
[[ "$NODE_RANK" == "0" || "$NODE_RANK" == "1" ]] || { echo "rank must be 0 or 1" >&2; exit 2; }

IMAGE="radixark/vllm-glm53-flash:sm121-v11-dflash2"
NAME="vllm_node"
MODEL_HOST_PATH="$MODEL_DIR"
MODEL_PATH="$MODEL_DIR/"
DRAFTER_HOST_PATH="${DRAFTER_HOST_PATH:-/var/tmp/models/GLM-5.3-Flash-DFlash2}"
CACHE_HOST_PATH="/var/tmp/glm53-vllm-cache"
HEAD_IP="${HEAD_IP:-192.168.102.1}"
MPORT="${MPORT:-29521}"
PORT="8000"

case "$NODE_RANK" in
  0) HOST_IP="${HOST_IP_RANK0:-192.168.102.1}"; HEADLESS="" ;;
  1) HOST_IP="${HOST_IP_RANK1:-192.168.102.2}"; HEADLESS="--headless" ;;
esac

test -f "$MODEL_HOST_PATH/config.json"
test -f "$DRAFTER_HOST_PATH/config.json"
test -f "$HOME/patches/sparse_attn_indexer_kpool.py"
mkdir -p "$CACHE_HOST_PATH"

# Wedge-safe replace: graceful stop first (never rm -f a GPU-active container — UVM wedge).
if docker ps --format '{{.Names}}' | grep -qx "$NAME"; then
  echo "gracefully stopping running $NAME ..."
  docker stop -t 30 "$NAME" >/dev/null || true
fi
docker rm "$NAME" 2>/dev/null || true

docker run --gpus all -d \
  --name "$NAME" --restart no \
  --memory=118g \
  --network host --ipc host --shm-size 32g \
  --ulimit memlock=-1:-1 --cap-add IPC_LOCK \
  --device /dev/infiniband:/dev/infiniband \
  -v /home/pcozz/models:/home/pcozz/models \
  -v "$CACHE_HOST_PATH:/cache" \
  -v "$CACHE_HOST_PATH/flashinfer:/root/.cache/flashinfer" \
  -v "$CACHE_HOST_PATH/tilelang:/root/.tilelang" \
  -e VLLM_HOST_IP=$HOST_IP \
  -e HF_HOME=/cache/huggingface \
  -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS="${VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS:-3600}" \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e TORCH_CUDA_ARCH_LIST=12.1a -e FLASHINFER_CUDA_ARCH_LIST=12.1a \
  -e FLASHINFER_DISABLE_VERSION_CHECK=1 \
  -e NCCL_NET=IB -e NCCL_IB_DISABLE=0 \
  -e NCCL_IB_HCA=rocep1s0f1 -e NCCL_IB_GID_INDEX=3 \
  -e NCCL_IB_ROCE_VERSION_NUM=2 -e NCCL_IB_ADDR_FAMILY=AF_INET \
  -e NCCL_IB_ADDR_RANGE=192.168.102.0/24 \
  -e NCCL_SOCKET_IFNAME=enp1s0f1np1 -e GLOO_SOCKET_IFNAME=enp1s0f1np1 \
  -e TP_SOCKET_IFNAME=enp1s0f1np1 -e MN_IF_NAME=enp1s0f1np1 \
  -e NCCL_NVLS_ENABLE=0 -e NCCL_CROSS_NIC=0 -e NCCL_IB_MERGE_NICS=0 \
  -e NCCL_CUMEM_ENABLE=0 -e NCCL_IGNORE_CPU_AFFINITY=1 -e NCCL_DEBUG=WARN \
  -e NCCL_BLOCKING_WAIT=0 \
  -e TORCH_NCCL_ASYNC_ERROR_HANDLING=1 \
  -e TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800 \
  -e TORCH_NCCL_DISABLE_WATCHDOG=1 \
  -v $HOME/patches/sparse_attn_indexer_kpool.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/sparse_attn_indexer_kpool.py:ro \
  -v $DRAFTER_HOST_PATH:/models/dflash2-draft:ro \
  "$IMAGE" \
  "$MODEL_PATH" \
  --served-model-name glm-5.3-flash \
  --host 0.0.0.0 --port "$PORT" \
  --trust-remote-code \
  --tensor-parallel-size 2 \
  --enable-expert-parallel \
  --gpu-memory-utilization "$GMU" \
  --max-model-len "$MAX_MODEL_LEN" \
  --max-num-seqs "$MAX_NUM_SEQS" --block-size "$BLOCK_SIZE" \
  --speculative-config '{"method":"dflash","model":"/models/dflash2-draft","num_speculative_tokens":'"$SPEC_NUM_TOKENS"'}' \
  --kv-cache-dtype fp8_e4m3 --kv-cache-memory "$KV_CACHE_MEM" \
  $EAGER_FLAG \
  "${GRAPH_ARGS[@]}" \
  --tool-call-parser glm47 --enable-auto-tool-choice \
  --reasoning-parser glm45 \
  --default-chat-template-kwargs '{"enable_thinking":true}' \
  --distributed-executor-backend mp \
  --nnodes 2 --node-rank "$NODE_RANK" \
  --master-addr "$HEAD_IP" --master-port "$MPORT" \
  $HEADLESS

echo "launched $NAME rank=$NODE_RANK host=$HOST_IP"
sleep 2
docker ps --format '{{.Names}} {{.Status}}' | grep "$NAME" || {
  echo "$NAME exited; inspect with: docker logs $NAME" >&2
  exit 1
}

The bind-mounted sparse_attn_indexer_kpool.py top-k patch is part of the authors' SM121 patch stack (available on request); the launcher tests for it before starting.

Labeled deltas — NOT the current production config:

@262K staging (behind the 33.29/42.01/80.98/121.81 table row above): MAX_MODEL_LEN=262144 KV_CACHE_MEM=3221225472 → pool 310,292 tokens.
Previous banked config (behind the published @1M table): GMU=0.795 + EAGER=1 (--enforce-eager), launched as MAX_MODEL_LEN=1048576 KV_CACHE_MEM=9663676416 bash launch_dflash2_tp2.sh <rank>.

Docker images

Upstream note (2026-09-03): GLM-5.3-Flash support landed in vLLM main (vllm-project/vllm#53906, merged 16:40 UTC). The pinned images below remain the validated serving stacks for this quant; upstream-main validation is a separate, queued leg.

Table with columns: Architecture, Image
Architecture	Image
SM90 (H100/H200)	`vllm/vllm-openai:glm53-flash-x86_64-cu130` (validated); `vllm/vllm-openai:nightly-x86_64` ≥ 2026-09-08 also boots this checkpoint (upstream PR #53906) — second arm, see H200 table
SM120 (RTX PRO 6000)	`cstechdev/vllm:glm53-flash-nope-sm120-cu130-20260826-r1`
SM121 (DGX Spark)	`radixark/vllm-glm53-flash:sm121-v11-dflash2` — no longer published (Docker Hub 404 as of 2026-09-11); the patched SM121 build recipe is maintained by the authors, available to collaborators on request

Quick start

bash
# 1. Download the weights (~178 GiB)
huggingface-cli download canada-quant/GLM-5.3-Flash-W4A16-MTP --local-dir /models/glm53-flash-w4a16-mtp

# 2. Serve (SM90 example — see recipes above)
docker run --gpus '"device=0,1,2,3"' --ipc=host --network=host --rm \
  -v /models:/models vllm/vllm-openai:glm53-flash-x86_64-cu130 \
  vllm serve /models/glm53-flash-w4a16-mtp --served-model-name glm53-w4 \
    --tensor-parallel-size 4 --enable-expert-parallel --max-num-seqs 512 \
    --max-model-len 262144 --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
    --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
    --trust-remote-code

# 3. Call it (OpenAI-compatible)
curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "glm53-w4",
  "messages": [{"role": "user", "content": "Prove there are infinitely many primes."}]
}'

Details

Table with columns: Field, Value
Field	Value
Base model	zai-org/GLM-5.3-Flash (BF16)
Architecture	`Glm5NextForConditionalGeneration` (`glm5_next`) — 45 decoder layers (+ MTP layer 45), hidden 4096, 288 routed experts (top-8) + 1 shared, MoE-intermediate 2048, KDA + DSA attention, 24-block vision tower, vocab 154,880
Weight quantization	W4A16, INT4, symmetric, group-size 128 (GPTQ, compressed-tensors `pack-quantized`), routed experts only
Quantized tensors	36,288 (42 MoE layers × 288 experts × 3 GEMMs) — verified exact
Kept in BF16	attention (KDA + DSA incl. indexer), dense prefix (layers 0–2), shared experts, router/gate, mHC tensors, embeddings, lm_head, norms, ,

vLLM serving note (config.json updated 2026-09-08 — re-download config.json if your copy predates this): vLLM's Glm5NextForConditionalGeneration matches quantization_config.ignore against its own fused module names (language_model.model.layers.N.mlp.gate_up_proj, language_model.lm_head, MTP model.layers.45.*), not the HF names llm-compressor wrote. The ignore list therefore carries both spellings plus the fused gate_up_proj entries and a prefix-agnostic re:.*\.layers\.45\..* for the MTP head. With an older config.json (765 entries, sha256 92ae6f37…) vLLM fails at load with KeyError: '…mlp.gate_up_proj.weight'; the fix is the current config.json only — weights are unchanged.

Validation

Static gates at build time (2026-08-27), all PASS:

exactly 36,288 packed tensors; zero compression outside routed experts;
vision key-set 348/348 identical to source, zero compressed, all dtypes match;
MTP layer present (889 keys), index rewrite purely additive;
dtype drift vs source: zero (282 pipeline-drifted tensors restored verbatim);
expert scale audit: no zero/collapsed scales.

Quantized on 8× NVIDIA B300, 2026-08-27. B300 smoke tests (transformers ≥5.16): loads as Glm5NextForConditionalGeneration (~110 s); text generation coherent; image captioning accurate.

Vision-quality note: the vision tower is BF16-passthrough but was not covered by the text-only calibration set. Image smoke tests pass; a vision eval leg (MMMU-class + OCR) is queued.

Research-stack known issue: DFlash2 admission wedge — root-caused across 21 legs; FIX VALIDATED (2026-09-04)

Affects the SM90 research stack only (x86_64, DFlash2 drafter n=7). The standard MTP serve path used by all production configs above is unaffected.

Symptom: prompts above a block-size-dependent bound are never admitted — Waiting: 1 / Running: 0, 0% KV, retry-forever until timeout (no crash, no OOM).
Measured bounds: block 2304 → wedges above ~15.5K tokens; block 1024 → ~17.4K; block 1536 → no wedge at all across the tested 16–24K class.
Mechanism (pinned): spec-decode admission requires a full-sequence reservation against the unified KV pool (1,012 blocks); the binding group is the drafter's SlidingWindowManager (block-16) — a 15,523-token request needs exactly 1,011 blocks (passes, zero margin), a ~15,556-token request needs 1,013 (wedges).
Root cause (pinned): the drafter SW manager allocates execution-side at full-sequence scale (cdiv(tokens,16)) while its true sliding-window need is only 130 blocks; the built-in skip-free mechanism fires but freed blocks are instantly re-consumed by the next full-sequence allocation.
Fix VALIDATED: cap the SW manager's execution allocation at min(cdiv(tokens, bs), cdiv(sliding_window, bs) + 2) (= 130 blocks) — serves 8,528 / 15,787 / 25,344 / 63,734 / 127,952 / 256,388 tokens, all PASS at the shipping block-2304 config (pre-fix wedge at ≥15,537), zero engine errors.
Quality spot-suite PASS: 24 deterministic wedge-class prompts, fixed stack vs spec-OFF control — 24/24 checks both arms, zero tokenization mismatch; no detectable quality impact at spot scope.
Workarounds without the patch: block 1536 serves the whole tested 16–24K class (its SW cap saturates) and holds the largest KV pool (1.12× of a 262K request vs 0.78× at block 2304).

SM121 kernel-level research findings (2026-09-09)

Custom-kernel program COMPLETE — the stock marlin remains the serving kernel. A pre-registered kernel-optimization program for the SM121 W4A16 MoE GEMM (the GEMM measured at 84.2% of the c1→c6 step-time delta; 0.3604 ms/tok/layer slope) executed to exhaustion: a custom persistent-ticket-scheduler marlin ("E-K2b") was bit-correct everywhere (0 output flips vs the no-scheduler reference across the full M sweep) but never met the ≤1.10 scheduling-overhead adoption gate (best ≈1.29 at M≥64), and all three follow-up axes — O1 claim-batching, O2 barrier-elision, O3 claim-ahead pipelining — were cleanly falsified (overheads 1.29–1.36, monotone worse). No custom kernel code ships; everything on this card runs the stock moe_wna16_marlin heuristic.
Upstream defect found and filed: vllm-project/vllm#56064. Forcing the marlin no-split-K (data-parallel, split_k=1) path produces a deterministic CUDA illegal memory access at M=256 on SM121/GB10 — clean at M≤128 on the same path, and the stock split-K-allowed call is clean at M=256. Reproduced on two independently built extension modules using the kernel's own determine_exec_config pick ⇒ the defect is in the shared marlin core's no-split-K path, not in caller scheduling. Serving is unaffected (the stock heuristic is the split-K path); this matters only to kernel researchers forcing DP/no-split-K execution (e.g. custom schedulers).
Own-drafter program ("dspark2"): specs + Phase-0 reference stack done; not trained. A zero-reuse (no radixark code) own drafter for this checkpoint is fully specced (12-step build order; SWA-2048, fp8 KV parity, K-adaptive draft depth, cudagraph-safe) with an 18-module stdlib reference implementation (237 unit tests green). No trained weights exist yet; the drafter behind every number on this card remains the radixark DFlash2 drafter. Program target: closing the mid-concurrency gap vs EXL3 documented above.

This section contributed via the Digby.ai coding harness from sha-pinned banked artifacts; reviewed by a human (the maintainer) before publication.

License

MIT, inherited from the base model. Follow the base model's usage terms.

Built, benchmarked, and documented with the Digby.ai coding harness, developed by CQL.ca.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

canada-quant

Model Tree

Base

zai-org/GLM-5.3-Flash

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

🏆 Headline — where W4A16 wins (all measured, same-harness comparisons)

−70% weights: 177.7 GiB vs ~599 GiB BF16 → 4 datacenter GPUs instead of 8, or a 2× DGX Spark desktop pair at full 1M context.
vs NVFP4 (LibertAI, strongest community quant) — 2× DGX Spark: W4A16 serves 1,048,576 tokens there (KV pool 1.30×); the NVFP4 checkpoint OOM'd 9/9 boots at any context on identical nodes (dmesg-verified). Throughput @262K: +53% seq1 / +101% seq6.
vs NVFP4 — H100 TP=4, matched MTP: wins c1 +4.8% · c8 +13.6% · saturated c32 +14.0% (vs NVFP4's best c32 config); mtp-off fixed-drafter c32 +22.4% — the batch edge is the marlin INT4 MoE GEMM, not the drafter.
vs EXL3 (MiaAI-Lab 2×-Spark recipe) — independent matched-protocol head-to-head, FINAL: W4A16 leads single-stream +10.4% (33.0 vs 29.9 tok/s) and long-prefill +31–39% (~1.2K vs 864–914 tok/s @300K-class); KV capacity 1.30× vs 1.20×. The vendor README overstated decode ~2.2× and prefill ~1.2× vs independent measurement.
Quality: AIME 2026 85.0% (102/120, n=120 final, max thinking) — beats the matched EXL3 arm 96/120 (80.0%) by +6 correct (+5.0 pt), FINAL 2026-09-06; AIME 2025 parity with NVFP4 on H100 (0.8833 vs 0.9000, 0.42σ); GSM8K 0.97; GPQA-Diamond within noise.
SM120 (Blackwell): throughput parity with NVFP4 (±0.7%) with MTP paying at every concurrency — +69.9% c1 / +37.6% c8 / +47.8% c32.
4× H200 (SM90, 141 GB) — measured 2026-09-08 on AWS p5en.48xlarge, published recipe as-is (TP=4, MTP N=2): c1 200 · c8 672 · c32 1,234 · c64 1,507 · c128 1,751 · c256 ~1,970 output tok/s (8192/1024, thinking on), KV pool 6.19M tokens ≈ 23 concurrent 262K requests. Beats the best H100 row at every concurrency (+9% c1 / +10% c8 / +6% c32) and keeps scaling to c256. Prefix caching ON measured no gain on H200 (−2…−5%). 8× H200 TP=8: 210 · 823 · 1,636 · 2,063 · 2,385 · 2,674 (c1…c256), TPOT 4.5 ms single-stream. 2× H200 TP=2: 262K context with a 9.8× KV pool (c32 814 tok/s) and 1M context serving-validated (929K-token prompt admitted and decoded on two GPUs).
Long context: single-stream decode is KV-length-insensitive to ≥486K (up to 357 tok/s, bf16 KV); long-prefill ceiling ~6.1–8.5K tok/s aggregate across the full 3-KV × c1–c32 matrix.
Research-stack bug found and fixed: a 21-leg investigation root-caused a hang-forever admission wedge in the DFlash2 spec-decode stack and validated a fix to 256,388-token prompts (24/24 quality spot-checks pass).

Why this quant exists

BF16 GLM-5.3-Flash needs eight 80 GB GPUs per replica; this artifact cuts weights to 177.7 GiB — four GPUs, or a dual DGX Spark desktop at 1M context.
Only the 36,288 routed-expert GEMMs are INT4 (symmetric, group-128, GPTQ); attention, router, shared experts, embeddings, the BF16 vision tower, and the BF16 MTP draft head (+40–50% single-stream) stay unquantized; canonical FP32 tensors preserved verbatim.
Not a new model — all capability comes from the base model.

Hardware

The model's context is 1M tokens (1,048,576). Each row is the largest context serving-validated on that config — x86 rows are KV-memory-bound, not model-bound.

Table with columns: Config, GPUs, Validated context, Serving stack
Config	GPUs	Validated context	Serving stack
DGX Spark GB10 (SM121), TP=2 + EP	2× 128 GB UMA	1M — KV pool 1,360,420 tok (1.30×)	DFlash2 drafter, fp8 KV — recipe below
RTX PRO 6000 (SM120), TP=4 + EP	4× 96 GB	512K (engine cap 524,288; 486K prompts measured)	MTP N=2, fp8 KV
H100 (SM90), TP=4 + EP	4× 80 GB	262K (256K prompts measured)	MTP N=2 or DFlash2, bf16 KV
H200 (SM90), TP=4 + EP	4× 141 GB

Quantization ran on 8× NVIDIA B300 (SM103) — see Quantization below; it is not a serving config.

Benchmarks vs competitors

Identical flags/prompts/hardware per row. Throughput = output tok/s, thinking ON.

vs EXL3 (MiaAI-Lab) — 2× DGX Spark, matched protocol, FINAL (2026-09-03)

EXL3 was brought to full serving at 1M context on the same hardware (KV pool 1,200,348 = 1.20×; W4A16 1,360,420 = 1.30×).
Like-for-like decode (TTFT excluded, 8K prompts, 256-token completions): W4A16 +10.4% c1 (33.0 vs 29.9 agg tok/s).
EXL3 leads mid-concurrency: c2 +67.4% (59.6 vs 35.6), c4 +89.1% (112.7 vs 59.6), c6 ≈+80.9% after correcting its max-num-seqs=4 scheduler cap; it sustains ~29 tok/s per stream at every concurrency (W4A16 steps down 33.0→11.2 seq1→6 on the 1M serve; the 262K serve holds ~20 at seq4+).
Saturated c32 (clean refire): EXL3 881–898 agg tok/s (32/32 streams, per-stream 27.4–28.1).
W4A16 +31–39% long-prefill (~1.2K tok/s vs 864–914 @300K-class prompts).
Vendor README overstated measured decode ~2.2× (claimed 62.9/51.7/37.1 per-stream c1/2/4 vs measured 26.6/21.2/17.0) and prefill ~1.2×.
Two mid-bench hardware deaths on the competitor node were quarantined and never banked; the c32 set was re-run clean.
Matched AIME-2026 quality arm: FINAL 2026-09-06 — W4A16 102/120 (85.0%) beats EXL3 96/120 (80.0%), +6 correct (+5.0 pt). EXL3 epochs 25/25/24/22, zero harness errors; 23/120 EXL3 rows hit the 131,072-token cap (W4A16: 16/18 misses were budget-exhaustion) — scored as written under the identical temp-0.0 / 131,072-token / n=120 config as our banked 85.0% arm.

4× H200 (SM90) — TP=4, published recipe, isl/osl 8192/1024, thinking ON (measured 2026-09-08, AWS p5en.48xlarge, image `glm53-flash-x86_64-cu130`)

Table with columns: config, c1, c8, c32, c64, c128, c256, MTP accept, TPOT p50 ms (c1 → c256)
config	c1	c8	c32	c64	c128	c256	MTP accept	TPOT p50 ms (c1 → c256)
MTP N=2, prefix caching OFF (published) — FINAL, alone, mean of 3 reps	195	698	1,258	1,529	1,789	1,954	53–54%	4.5–4.9 / 10 / 20 / 38 / 66 / 118 — GSM8K-200 on this recipe on H200: 0.99

vs NVFP4 — H100/H200 (SM90), TP=4, MTP both sides, isl/osl 8192/1024

Table with columns: concurrency, W4A16 MTP N=2 (this), NVFP4 MTP N=2, NVFP4 MTP N=5 (its best c32 config)
concurrency	W4A16 MTP N=2 (this)	NVFP4 MTP N=2	NVFP4 MTP N=5 (its best c32 config)
1	183.89	180.96	175.44
8	249.94	251.14	220.05
32	1161.13 †	770.66	1018.13
MTP acceptance	52–55%	~52%

Headline: +4.8% c1 / +13.6% c8 / +14.0% saturated c32 vs NVFP4's banked N=5 config.
† c=32 with --gpu-memory-utilization 0.92 (KV-pool capacity): 1161.13 tok/s (independent tuned run: 1145.05). The c=32 lever is KV-pool size, not a kernel gap.
Fixed-drafter 2×2 re-confirm (both arms on the corrected #54374-port overlay): matched c1/c8 within noise (180.22/254.39 vs 182.13/252.89); saturated c32 win holds (+18.1%, 804.02 vs 680.99).
MTP-off on both: W4A16 leads +2.7% / +10.5% / +22.4% at c1/c8/c32 — the c32 advantage is the marlin INT4 MoE GEMM at batch, not an MTP artifact.
MTP adds +41.1%/+14.3%/−4.0% (W4A16) and +46.4%/+25.6%/−0.4% (NVFP4) at c1/c8/c32 on SM90.

vs NVFP4 — RTX PRO 6000 (SM120, Blackwell), TP=4, matched MTP N=2

Table with columns: concurrency, W4A16 (this), NVFP4, Δ
concurrency	W4A16 (this)	NVFP4	Δ
1	109.90	109.34	+0.5%
8	318.54	319.46	−0.3%
32	534.37	530.75	+0.7%

Parity within noise (±0.7%); verbatim-methodology re-grid: c1 106.97 (−2.7%, 8-prompt high-variance class), c8 320.51 (+0.6%), c32 539.63 (+1.0%), MTP acceptance 50–53% — no stack drift.
MTP pays at every concurrency on SM120: +69.9% c1 (106.97 vs 62.96), +37.6% c8 (320.51 vs 232.89), +47.8% c32 (539.63 vs 365.02) — a sign flip vs SM90 (drafter costs ~4% at saturated c32 there). Keep MTP on at every concurrency on SM120.
KV-dtype at batch: fp8 vs bf16 costs nothing at c1/c8 (±0.05%) and ≈4.7% at c32 (365.02 vs 382.07) — lever ranking MTP ≫ KV dtype. (auto is the working bf16-KV path; bf16/bfloat16 variants fail on this stack.)

Long-context decode at batch (SM120, TP=4, 524K ctx, MTP N=2 — decode-isolated, TTFT excluded)

Per-stream decode tok/s (full-contention cohort bands — serialized prefills split stages into early/late cohorts, so bands, not blended medians):

Table with columns: KV length, KV dtype, c1, c4, c8, c32
KV length	KV dtype	c1	c4	c8	c32
8K (aggregate grid)	fp8	106.97	—	40.06	16.86
138.6K	fp8	143.94	2.92–7.53	2.49–3.74	2.24–2.72
138.6K	bf16 (`auto`)

Single-stream decode is KV-length-insensitive to ≥486K (143.94 → 177.15 → 357.34 tok/s at 138.6K/277K/486K bf16 — rises, if anything).
At-batch collapse saturates with KV length: ~13× per-stream drop at c8 vs the 8K grid, then flat ~2.2–3.7 tok/s/stream at c4–c32 from 138.6K through 486K — mechanism is target-model forward compute at batch × long-KV (~4.4M tokens of live KV at c32), not the drafter.
KV dtype is a c1-only ~5% lever (bf16 +5.7% @138.6K); at-batch bands overlap fp8/bf16 — the long-KV-amplifies-fp8-cost hypothesis is refuted.
MTP acceptance never collapses: c1 ≈80–86% → c8 73.9% (n=1,286) → c32 71.6% (n=4,714) at 138.6K; 60.0% single-stream at 486K (small sample, disclosed). The saturated c32 stage decoded 2.53 tok/s/stream while the drafter accepted 71.6% — conclusively not acceptance-driven.
At-batch lever ranking FINAL: MTP drafter ≫ everything measured; KV dtype ≈ c1-only ~5%; drafter acceptance is not the bottleneck.

Long-prefill at batch (SM120) — full 3-KV × c1–c32 matrix COMPLETE

Single-stream prefill: ~6,100–7,100 tok/s for 277K–486K prompts (TTFT first request 39.2–79.3 s).
At batch, long prefills are essentially serialized (stage wall ≈ N × single-stream time).
Aggregate prefill ceiling ~6.1–8.5K tok/s — KV-length-insensitive AND batch-insensitive:
138.6K: 6,735 (c1) → 7,134 (c8) → 7,897 (c16) → 8,486 (c32) tok/s.
277K: 5,211 (c1) → 7,177 (c8) → 7,784 (c16) → 7,357 (c32) tok/s.
486K: 6,136 (c1) → 6,892 (c8) → 7,483 (c16) → 6,988.6 (c32) tok/s.
Serialization ratios (wall / N×c1): 138.6K 0.94/0.85/0.79 · 277K 0.73/0.67/0.71 · 486K 0.89/0.82/0.88 (c8/c16/c32) — deeper-KV rows behave closer to strict serialization.
mnb lever fully mapped: 8192 ✓ (validated production config) → 16384 ✓ (validated fallback, ceiling within ~1% at both 277K and 486K) → 32768 ✗ (CUDA OOM at 486K — 3.98 GiB needed vs 1.70 GiB free at mem-util 0.95). Ceiling is chunk-size-insensitive → attention/forward compute-bound, not scheduling-bound.
Operational guidance: admission pacing / prefill scheduling is the only lever that moves long-context batch UX (mnb tuning and MTP do not address prefill serialization).

2× DGX Spark GB10 (SM121), TP=2, DFlash2 block-diffusion spec-decode

8K prompts, 256-token completions, aggregate tok/s:

Table with columns: config, seq1, seq2, seq4, seq6
config	seq1	seq2	seq4	seq6
W4A16+DFlash2 @262K (this)	33.29	42.01	80.98	121.81
NVFP4-MTP4 reference	21.8	41	66	60.5
Δ	+53%	+2.4%	+23%

256K-token single prompt on the 1M serve: 30.85 tok/s decode (ttft 5.8 s).
Structural: the NVFP4 checkpoint (90.67 GiB/node weights vs ~30 for W4A16) OOM'd all 9 boot attempts across 1M/300K/262K on the 121 GiB unified-memory nodes (dmesg-verified) — it cannot serve at any usable context on 2× Spark.

Quality

AIME 2026 (30 problems, max thinking, 131,072-token budget, DFlash2 stack, 2× Spark): 102/120 = 85.0% — n=120 FINAL. Epochs 28/30, 25/30, 26/30, 23/30; 16/18 misses are budget-exhaustion truncations, 2/18 genuine wrong answers. The LibertAI NVFP4 external AIME-2025 reference (0.9000) remains +5.0 points ahead; the matched AIME-2026 EXL3 arm is FINAL 2026-09-06: 96/120 (80.0%) — W4A16 wins by +6 correct (+5.0 pt).
AIME 2025 n=120, 4×30, max thinking, 131,072 budget — H100 (SM90): parity — W4A16 0.8833 (106/120) vs NVFP4 0.9000 (0.42σ). An earlier 0.7833 read was a serving-stack drafter-corruption bug (upstream vLLM #54374), since fixed.
AIME 2025 — RTX PRO 6000 (SM120): raw-arm deficit real (~2σ; 0.8083/0.8000 pooled vs 0.9000), mechanism decomposed and fix VALIDATED. ≈63% of the deficit is a mechanical budget-wall (EMPTY-answer rate 11.7–14.2% vs NVFP4's 3.3%; doubling the budget recovered +5.8 pt); ≈37% is SM120 kernel numerics (near-tie argmax flips vs SM90, logit-quantified).
Fix stack VALIDATED (zero GPU cost): whole-trace commit hook + thinking-budget force-commit (98,304-token cap) lifts the score to 0.8833 two-way / 0.8917 hook-credited — within noise of NVFP4 0.9000 (0.42σ / 0.21σ).
Production-overlay pilot passed (committed 0.8917, zero modified non-empty responses across 120 requests); production adoption remains gated on the eval sign-off chain (owner decision pending).
GSM8K: parity — 0.9704–0.9750 across all three quants on both x86 rigs.
GPQA-Diamond n=198 @131k: 0.8586 vs 0.8687 NVFP4 (SM90), 0.8586 vs 0.8737 (SM120) — within noise.
Matched AIME-2026 single-epoch arms on x86 (n=30, max thinking): W4A16 24/30 (RTX PRO 6000) and 22/30 (H100) vs NVFP4 23/30 — within noise at n=30.

Serving

The same checkpoint serves on every NVIDIA architecture from Hopper through Blackwell. Each recipe is the exact config the benchmarks ran.

SM90 — H100 / H200 (recommended: TP=4)

Re-download config.json if your copy predates 2026-09-08 (commit ece4697): older copies fail in vLLM with KeyError: 'layers.0.mlp.gate_up_proj.weight' — see the vLLM serving note under Details. Weights are unchanged.

bash
docker run --gpus '"device=0,1,2,3"' --ipc=host --network=host --rm \
  -v /models:/models vllm/vllm-openai:glm53-flash-x86_64-cu130 \
  vllm serve /models/glm53-flash-w4a16-mtp \
    --served-model-name glm53-w4 \
    --tensor-parallel-size 4 --enable-expert-parallel \
    --max-model-len 262144 --max-num-seqs 512 \
    --block-size 64 --gpu-memory-utilization 0.92 --no-enable-prefix-caching \
    --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
    --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
    --trust-remote-code --port 8000

MTP num_speculative_tokens: 2 is the sweet spot on Hopper (52–55% acceptance, best c1/c8); N=5 collapses acceptance (52→30%). H200 N=1 / N=3 / MTP-off and TP=8 rows are in flight (2026-09-08).
H200 (141 GB) with the recipe above, unchanged — engaged: Using 'MARLIN' WNA16 MoE backend, FLASH_ATTN_MLA_SPARSE, SpeculativeConfig(method='mtp', num_spec_tokens=2), attention block auto-set to 1152 tokens (mamba page alignment; --block-size 64 is inert), GPU KV cache size: 6,194,528 tokens (76.5 GiB free after the 45.25 GiB/GPU weights). Prefix caching ON measured −2…−5% vs OFF at every concurrency → keep --no-enable-prefix-caching. Optional for c≳170 with N=2: --max-cudagraph-capture-size 1536 keeps large decode batches inside full CUDA graphs (default cap 512 tokens/step; costs ~0.4M tokens of KV). Same recipe also boots on upstream vllm/vllm-openai:nightly-x86_64 (v0.28.1rc1.dev516+, includes PR #53906): FLASHINFER_MLA_SPARSE_SM90 backend, ~6.06M-token pool, −22% c1 / −6% c32 / tie at c64–c128 vs the pinned image (and that backend IMAs on ≥131K prompts — pass --attention-backend FLASH_ATTN_MLA_SPARSE on the nightly).

SM120 — RTX PRO 6000 (Blackwell)

Image cstechdev/vllm:glm53-flash-nope-sm120-cu130-20260826-r1 (SM120-patched build).

bash
docker run --gpus '"device=4,5,6,7"' --ipc=host --network=host --rm \
  -v /models:/models cstechdev/vllm:glm53-flash-nope-sm120-cu130-20260826-r1 \
  vllm serve /models/glm53-flash-w4a16-mtp \
    --served-model-name glm53-w4 \
    --tensor-parallel-size 4 --enable-expert-parallel \
    --max-model-len 262144 --max-num-seqs 64 --max-num-batched-tokens 8192 \
    --kv-cache-dtype fp8 --gpu-memory-utilization 0.92 --enable-prefix-caching \
    --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
    --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
    --trust-remote-code --port 8001

KV fp8 is required at 262K on 96 GB cards; --max-num-seqs 64 is bench-validated (48/36 fallback ladder); MTP N=2 matches NVFP4 to ±0.7%.

SM121 — 2× DGX Spark GB10 (desktop, 1M context)

Production launch (current):

bash
# rank1 (worker) FIRST → wait 25 s → rank0 (head):
MAX_MODEL_LEN=1048576 KV_CACHE_MEM=9663676416 GMU=0.90 EAGER=0 GRAPHS=1 bash launch_dflash2_tp2.sh <rank>

Engine args the launcher produces (weights bind-mounted at /models/glm-5.3-w4a16-mtp; set MODEL_DIR to your local weights path):

bash
vllm serve /models/glm-5.3-w4a16-mtp/ \
  --served-model-name glm-5.3-flash --host 0.0.0.0 --port 8000 --trust-remote-code \
  --tensor-parallel-size 2 --enable-expert-parallel \
  --gpu-memory-utilization 0.90 \
  --max-model-len 1048576 --max-num-seqs 6 --block-size 2304 \
  --speculative-config '{"method":"dflash","model":"/models/dflash2-draft","num_speculative_tokens":7}' \
  --kv-cache-dtype fp8_e4m3 --kv-cache-memory 9663676416 \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[1,2,4,8,16,24,32]}' \
  --tool-call-parser glm47 --enable-auto-tool-choice --reasoning-parser glm45 \
  --default-chat-template-kwargs '{"enable_thinking":true}' \
  --distributed-executor-backend mp --nnodes 2 --node-rank <0|1> \
  --master-addr <HEAD_IP> --master-port 29521    # + --headless on rank1

Stack + topology:

Image note (2026-09-11): the pinned SM121 image radixark/vllm-glm53-flash:sm121-v11-dflash2 is no longer published — its Docker Hub repo returns 404 as of 2026-09-11. Our fleet runs a locally cached copy (an aarch64/CUDA-13.0 SM121 vLLM build carrying the DFlash2 serving patches; the stock upstream DFlash2 image cannot boot this drafter — see the defect section below). Until an upstream stack lands, the reproducible paths are: (a) upstream vLLM main (≥ vllm-project/vllm#53906) serves these weights on Spark with MTP — without DFlash2; (b) the exact patched SM121 build recipe is maintained by the authors and is available to collaborators on request.
Drafter incoai/GLM-5.3-Flash-DFlash2 mounted at /models/dflash2-draft; weights = this repo (178 GiB, bind-mounted into the container; config.json must be the current published one (the ignore list carries both HF and vLLM module spellings; older 765-entry copies fail to load)).
Drafter internals: block 8, selector_rank 256, top_k 16, layers [5,14,24,33,42]; 7 speculative tokens (any other count boot-wedges).
KV pool: 1,360,420 tokens fp8_e4m3 = 1.30× a full 1M request.
RoCE fabric: NCCL_IB_HCA=rocep1s0f1, GID index 3, RoCEv2, 192.168.102.0/24 (enp1s0f1np1), — path at 97.98 Gbps line-rate (direct-attach measured worse).

Full launcher (verbatim, sha256 440ca004b719…):

bash
#!/usr/bin/env bash
# GLM-5.3-Flash W4A16 + DFlash2 speculative decoding — TP2 across two DGX Spark GB10 (SM121) nodes.
# Adapted from tonyd2wild/GLM-5.3-Flash-NVFP4-DFlash2-2x-DGX-Spark/launch-glm53-vllm-tp2-dflash2.sh
# (their 46.9 tok/s config) for a two-Spark RoCE fabric:
#   - W4A16 weights (/home/pcozz/models/glm-5.3-w4a16-mtp/) instead of NVFP4
#   - fabric: rocep1s0f1 / enp1s0f1np1 / 192.168.102.0/24
#   - proven W4A16 serve args (EP on, fp8_e4m3 KV, block 2304, seqs 6)
#   - wedge-safety: graceful docker stop -t 30 (NEVER rm -f a GPU-active container), --memory=118g cap
# Usage: launch_dflash2_tp2.sh <0|1>  — worker (1) FIRST, wait 25s, then head (0).
# Env knobs (defaults = 262K serve):
#   MAX_MODEL_LEN  (default 262144; 1048576 = architecture max, needs KV_CACHE_MEM raise)
#   KV_CACHE_MEM   (default 3221225472 = 3 GiB -> 310,292 tok pool, ~10.38 KB/tok fp8;
#                   1M pool needs ~10.9 GiB -> use 11811160064 = 11 GiB -> ~1.09M tok;
#                   safer 1M value: 9663676416 = 9 GiB -> ~1.36M)
#   MODEL_DIR      (default /home/pcozz/models/glm-5.3-w4a16-mtp = W4A16)
set -euo pipefail
MAX_MODEL_LEN="${MAX_MODEL_LEN:-262144}"
KV_CACHE_MEM="${KV_CACHE_MEM:-3221225472}"
MAX_NUM_SEQS="${MAX_NUM_SEQS:-6}"
MODEL_DIR="${MODEL_DIR:-/home/pcozz/models/glm-5.3-w4a16-mtp}"
# DFlash2 drafter lever: override num_speculative_tokens (default 7; any other count boot-wedges).
SPEC_NUM_TOKENS="${SPEC_NUM_TOKENS:-7}"
# EAGER=1 (default) = --enforce-eager; current production passes EAGER=0 GRAPHS=1 (CUDA graphs ON).
EAGER="${EAGER:-1}"
if [ "$EAGER" = "1" ]; then EAGER_FLAG="--enforce-eager"; else EAGER_FLAG=""; fi
# Note: engine may raise block-size to >= mamba page size (see boot log).
BLOCK_SIZE="${BLOCK_SIZE:-2304}"
# Banked default 0.795; current production passes GMU=0.90.
GMU="${GMU:-0.795}"
# GRAPHS=1 + EAGER=0 = FULL_AND_PIECEWISE capture sizes 1-32 (current production).
GRAPH_ARGS=()
if [ "${GRAPHS:-0}" = "1" ]; then GRAPH_ARGS=(--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[1,2,4,8,16,24,32]}'); fi

NODE_RANK="${1:?usage: launch_dflash2_tp2.sh <0|1>}"
[[ "$NODE_RANK" == "0" || "$NODE_RANK" == "1" ]] || { echo "rank must be 0 or 1" >&2; exit 2; }

IMAGE="radixark/vllm-glm53-flash:sm121-v11-dflash2"
NAME="vllm_node"
MODEL_HOST_PATH="$MODEL_DIR"
MODEL_PATH="$MODEL_DIR/"
DRAFTER_HOST_PATH="${DRAFTER_HOST_PATH:-/var/tmp/models/GLM-5.3-Flash-DFlash2}"
CACHE_HOST_PATH="/var/tmp/glm53-vllm-cache"
HEAD_IP="${HEAD_IP:-192.168.102.1}"
MPORT="${MPORT:-29521}"
PORT="8000"

case "$NODE_RANK" in
  0) HOST_IP="${HOST_IP_RANK0:-192.168.102.1}"; HEADLESS="" ;;
  1) HOST_IP="${HOST_IP_RANK1:-192.168.102.2}"; HEADLESS="--headless" ;;
esac

test -f "$MODEL_HOST_PATH/config.json"
test -f "$DRAFTER_HOST_PATH/config.json"
test -f "$HOME/patches/sparse_attn_indexer_kpool.py"
mkdir -p "$CACHE_HOST_PATH"

# Wedge-safe replace: graceful stop first (never rm -f a GPU-active container — UVM wedge).
if docker ps --format '{{.Names}}' | grep -qx "$NAME"; then
  echo "gracefully stopping running $NAME ..."
  docker stop -t 30 "$NAME" >/dev/null || true
fi
docker rm "$NAME" 2>/dev/null || true

docker run --gpus all -d \
  --name "$NAME" --restart no \
  --memory=118g \
  --network host --ipc host --shm-size 32g \
  --ulimit memlock=-1:-1 --cap-add IPC_LOCK \
  --device /dev/infiniband:/dev/infiniband \
  -v /home/pcozz/models:/home/pcozz/models \
  -v "$CACHE_HOST_PATH:/cache" \
  -v "$CACHE_HOST_PATH/flashinfer:/root/.cache/flashinfer" \
  -v "$CACHE_HOST_PATH/tilelang:/root/.tilelang" \
  -e VLLM_HOST_IP=$HOST_IP \
  -e HF_HOME=/cache/huggingface \
  -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS="${VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS:-3600}" \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e TORCH_CUDA_ARCH_LIST=12.1a -e FLASHINFER_CUDA_ARCH_LIST=12.1a \
  -e FLASHINFER_DISABLE_VERSION_CHECK=1 \
  -e NCCL_NET=IB -e NCCL_IB_DISABLE=0 \
  -e NCCL_IB_HCA=rocep1s0f1 -e NCCL_IB_GID_INDEX=3 \
  -e NCCL_IB_ROCE_VERSION_NUM=2 -e NCCL_IB_ADDR_FAMILY=AF_INET \
  -e NCCL_IB_ADDR_RANGE=192.168.102.0/24 \
  -e NCCL_SOCKET_IFNAME=enp1s0f1np1 -e GLOO_SOCKET_IFNAME=enp1s0f1np1 \
  -e TP_SOCKET_IFNAME=enp1s0f1np1 -e MN_IF_NAME=enp1s0f1np1 \
  -e NCCL_NVLS_ENABLE=0 -e NCCL_CROSS_NIC=0 -e NCCL_IB_MERGE_NICS=0 \
  -e NCCL_CUMEM_ENABLE=0 -e NCCL_IGNORE_CPU_AFFINITY=1 -e NCCL_DEBUG=WARN \
  -e NCCL_BLOCKING_WAIT=0 \
  -e TORCH_NCCL_ASYNC_ERROR_HANDLING=1 \
  -e TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800 \
  -e TORCH_NCCL_DISABLE_WATCHDOG=1 \
  -v $HOME/patches/sparse_attn_indexer_kpool.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/sparse_attn_indexer_kpool.py:ro \
  -v $DRAFTER_HOST_PATH:/models/dflash2-draft:ro \
  "$IMAGE" \
  "$MODEL_PATH" \
  --served-model-name glm-5.3-flash \
  --host 0.0.0.0 --port "$PORT" \
  --trust-remote-code \
  --tensor-parallel-size 2 \
  --enable-expert-parallel \
  --gpu-memory-utilization "$GMU" \
  --max-model-len "$MAX_MODEL_LEN" \
  --max-num-seqs "$MAX_NUM_SEQS" --block-size "$BLOCK_SIZE" \
  --speculative-config '{"method":"dflash","model":"/models/dflash2-draft","num_speculative_tokens":'"$SPEC_NUM_TOKENS"'}' \
  --kv-cache-dtype fp8_e4m3 --kv-cache-memory "$KV_CACHE_MEM" \
  $EAGER_FLAG \
  "${GRAPH_ARGS[@]}" \
  --tool-call-parser glm47 --enable-auto-tool-choice \
  --reasoning-parser glm45 \
  --default-chat-template-kwargs '{"enable_thinking":true}' \
  --distributed-executor-backend mp \
  --nnodes 2 --node-rank "$NODE_RANK" \
  --master-addr "$HEAD_IP" --master-port "$MPORT" \
  $HEADLESS

echo "launched $NAME rank=$NODE_RANK host=$HOST_IP"
sleep 2
docker ps --format '{{.Names}} {{.Status}}' | grep "$NAME" || {
  echo "$NAME exited; inspect with: docker logs $NAME" >&2
  exit 1
}

The bind-mounted sparse_attn_indexer_kpool.py top-k patch is part of the authors' SM121 patch stack (available on request); the launcher tests for it before starting.

Labeled deltas — NOT the current production config:

@262K staging (behind the 33.29/42.01/80.98/121.81 table row above): MAX_MODEL_LEN=262144 KV_CACHE_MEM=3221225472 → pool 310,292 tokens.
Previous banked config (behind the published @1M table): GMU=0.795 + EAGER=1 (--enforce-eager), launched as MAX_MODEL_LEN=1048576 KV_CACHE_MEM=9663676416 bash launch_dflash2_tp2.sh <rank>.

Docker images

Upstream note (2026-09-03): GLM-5.3-Flash support landed in vLLM main (vllm-project/vllm#53906, merged 16:40 UTC). The pinned images below remain the validated serving stacks for this quant; upstream-main validation is a separate, queued leg.

Table with columns: Architecture, Image
Architecture	Image
SM90 (H100/H200)	`vllm/vllm-openai:glm53-flash-x86_64-cu130` (validated); `vllm/vllm-openai:nightly-x86_64` ≥ 2026-09-08 also boots this checkpoint (upstream PR #53906) — second arm, see H200 table
SM120 (RTX PRO 6000)	`cstechdev/vllm:glm53-flash-nope-sm120-cu130-20260826-r1`
SM121 (DGX Spark)	`radixark/vllm-glm53-flash:sm121-v11-dflash2` — no longer published (Docker Hub 404 as of 2026-09-11); the patched SM121 build recipe is maintained by the authors, available to collaborators on request

Quick start

bash
# 1. Download the weights (~178 GiB)
huggingface-cli download canada-quant/GLM-5.3-Flash-W4A16-MTP --local-dir /models/glm53-flash-w4a16-mtp

# 2. Serve (SM90 example — see recipes above)
docker run --gpus '"device=0,1,2,3"' --ipc=host --network=host --rm \
  -v /models:/models vllm/vllm-openai:glm53-flash-x86_64-cu130 \
  vllm serve /models/glm53-flash-w4a16-mtp --served-model-name glm53-w4 \
    --tensor-parallel-size 4 --enable-expert-parallel --max-num-seqs 512 \
    --max-model-len 262144 --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
    --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
    --trust-remote-code

# 3. Call it (OpenAI-compatible)
curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "glm53-w4",
  "messages": [{"role": "user", "content": "Prove there are infinitely many primes."}]
}'

Details

Table with columns: Field, Value
Field	Value
Base model	zai-org/GLM-5.3-Flash (BF16)
Architecture	`Glm5NextForConditionalGeneration` (`glm5_next`) — 45 decoder layers (+ MTP layer 45), hidden 4096, 288 routed experts (top-8) + 1 shared, MoE-intermediate 2048, KDA + DSA attention, 24-block vision tower, vocab 154,880
Weight quantization	W4A16, INT4, symmetric, group-size 128 (GPTQ, compressed-tensors `pack-quantized`), routed experts only
Quantized tensors	36,288 (42 MoE layers × 288 experts × 3 GEMMs) — verified exact
Kept in BF16	attention (KDA + DSA incl. indexer), dense prefix (layers 0–2), shared experts, router/gate, mHC tensors, embeddings, lm_head, norms, ,

vLLM serving note (config.json updated 2026-09-08 — re-download config.json if your copy predates this): vLLM's Glm5NextForConditionalGeneration matches quantization_config.ignore against its own fused module names (language_model.model.layers.N.mlp.gate_up_proj, language_model.lm_head, MTP model.layers.45.*), not the HF names llm-compressor wrote. The ignore list therefore carries both spellings plus the fused gate_up_proj entries and a prefix-agnostic re:.*\.layers\.45\..* for the MTP head. With an older config.json (765 entries, sha256 92ae6f37…) vLLM fails at load with KeyError: '…mlp.gate_up_proj.weight'; the fix is the current config.json only — weights are unchanged.

Validation

Static gates at build time (2026-08-27), all PASS:

exactly 36,288 packed tensors; zero compression outside routed experts;
vision key-set 348/348 identical to source, zero compressed, all dtypes match;
MTP layer present (889 keys), index rewrite purely additive;
dtype drift vs source: zero (282 pipeline-drifted tensors restored verbatim);
expert scale audit: no zero/collapsed scales.

Quantized on 8× NVIDIA B300, 2026-08-27. B300 smoke tests (transformers ≥5.16): loads as Glm5NextForConditionalGeneration (~110 s); text generation coherent; image captioning accurate.

Vision-quality note: the vision tower is BF16-passthrough but was not covered by the text-only calibration set. Image smoke tests pass; a vision eval leg (MMMU-class + OCR) is queued.

Research-stack known issue: DFlash2 admission wedge — root-caused across 21 legs; FIX VALIDATED (2026-09-04)

Affects the SM90 research stack only (x86_64, DFlash2 drafter n=7). The standard MTP serve path used by all production configs above is unaffected.

Symptom: prompts above a block-size-dependent bound are never admitted — Waiting: 1 / Running: 0, 0% KV, retry-forever until timeout (no crash, no OOM).
Measured bounds: block 2304 → wedges above ~15.5K tokens; block 1024 → ~17.4K; block 1536 → no wedge at all across the tested 16–24K class.
Mechanism (pinned): spec-decode admission requires a full-sequence reservation against the unified KV pool (1,012 blocks); the binding group is the drafter's SlidingWindowManager (block-16) — a 15,523-token request needs exactly 1,011 blocks (passes, zero margin), a ~15,556-token request needs 1,013 (wedges).
Root cause (pinned): the drafter SW manager allocates execution-side at full-sequence scale (cdiv(tokens,16)) while its true sliding-window need is only 130 blocks; the built-in skip-free mechanism fires but freed blocks are instantly re-consumed by the next full-sequence allocation.
Fix VALIDATED: cap the SW manager's execution allocation at min(cdiv(tokens, bs), cdiv(sliding_window, bs) + 2) (= 130 blocks) — serves 8,528 / 15,787 / 25,344 / 63,734 / 127,952 / 256,388 tokens, all PASS at the shipping block-2304 config (pre-fix wedge at ≥15,537), zero engine errors.
Quality spot-suite PASS: 24 deterministic wedge-class prompts, fixed stack vs spec-OFF control — 24/24 checks both arms, zero tokenization mismatch; no detectable quality impact at spot scope.
Workarounds without the patch: block 1536 serves the whole tested 16–24K class (its SW cap saturates) and holds the largest KV pool (1.12× of a 262K request vs 0.78× at block 2304).

SM121 kernel-level research findings (2026-09-09)

Custom-kernel program COMPLETE — the stock marlin remains the serving kernel. A pre-registered kernel-optimization program for the SM121 W4A16 MoE GEMM (the GEMM measured at 84.2% of the c1→c6 step-time delta; 0.3604 ms/tok/layer slope) executed to exhaustion: a custom persistent-ticket-scheduler marlin ("E-K2b") was bit-correct everywhere (0 output flips vs the no-scheduler reference across the full M sweep) but never met the ≤1.10 scheduling-overhead adoption gate (best ≈1.29 at M≥64), and all three follow-up axes — O1 claim-batching, O2 barrier-elision, O3 claim-ahead pipelining — were cleanly falsified (overheads 1.29–1.36, monotone worse). No custom kernel code ships; everything on this card runs the stock moe_wna16_marlin heuristic.
Upstream defect found and filed: vllm-project/vllm#56064. Forcing the marlin no-split-K (data-parallel, split_k=1) path produces a deterministic CUDA illegal memory access at M=256 on SM121/GB10 — clean at M≤128 on the same path, and the stock split-K-allowed call is clean at M=256. Reproduced on two independently built extension modules using the kernel's own determine_exec_config pick ⇒ the defect is in the shared marlin core's no-split-K path, not in caller scheduling. Serving is unaffected (the stock heuristic is the split-K path); this matters only to kernel researchers forcing DP/no-split-K execution (e.g. custom schedulers).
Own-drafter program ("dspark2"): specs + Phase-0 reference stack done; not trained. A zero-reuse (no radixark code) own drafter for this checkpoint is fully specced (12-step build order; SWA-2048, fp8 KV parity, K-adaptive draft depth, cudagraph-safe) with an 18-module stdlib reference implementation (237 unit tests green). No trained weights exist yet; the drafter behind every number on this card remains the radixark DFlash2 drafter. Program target: closing the mid-concurrency gap vs EXL3 documented above.

This section contributed via the Digby.ai coding harness from sha-pinned banked artifacts; reviewed by a human (the maintainer) before publication.

License

MIT, inherited from the base model. Follow the base model's usage terms.

Built, benchmarked, and documented with the Digby.ai coding harness, developed by CQL.ca.

glm-5.3-w4a16-mtp

README

🏆 Headline — where W4A16 wins (all measured, same-harness comparisons)

Why this quant exists

Hardware

Benchmarks vs competitors

vs EXL3 (MiaAI-Lab) — 2× DGX Spark, matched protocol, FINAL (2026-09-03)

4× H200 (SM90) — TP=4, published recipe, isl/osl 8192/1024, thinking ON (measured 2026-09-08, AWS p5en.48xlarge, image glm53-flash-x86_64-cu130)

vs NVFP4 — H100/H200 (SM90), TP=4, MTP both sides, isl/osl 8192/1024

vs NVFP4 — RTX PRO 6000 (SM120, Blackwell), TP=4, matched MTP N=2

Long-context decode at batch (SM120, TP=4, 524K ctx, MTP N=2 — decode-isolated, TTFT excluded)

Long-prefill at batch (SM120) — full 3-KV × c1–c32 matrix COMPLETE

2× DGX Spark GB10 (SM121), TP=2, DFlash2 block-diffusion spec-decode

Quality

Serving

SM90 — H100 / H200 (recommended: TP=4)

SM120 — RTX PRO 6000 (Blackwell)

SM121 — 2× DGX Spark GB10 (desktop, 1M context)

Docker images

Quick start

Details

Validation

Research-stack known issue: DFlash2 admission wedge — root-caused across 21 legs; FIX VALIDATED (2026-09-04)

SM121 kernel-level research findings (2026-09-09)

License

Explore FriendliAI today

README

🏆 Headline — where W4A16 wins (all measured, same-harness comparisons)

Why this quant exists

Hardware

Benchmarks vs competitors

vs EXL3 (MiaAI-Lab) — 2× DGX Spark, matched protocol, FINAL (2026-09-03)

4× H200 (SM90) — TP=4, published recipe, isl/osl 8192/1024, thinking ON (measured 2026-09-08, AWS p5en.48xlarge, image glm53-flash-x86_64-cu130)

vs NVFP4 — H100/H200 (SM90), TP=4, MTP both sides, isl/osl 8192/1024

vs NVFP4 — RTX PRO 6000 (SM120, Blackwell), TP=4, matched MTP N=2

Long-context decode at batch (SM120, TP=4, 524K ctx, MTP N=2 — decode-isolated, TTFT excluded)

Long-prefill at batch (SM120) — full 3-KV × c1–c32 matrix COMPLETE

2× DGX Spark GB10 (SM121), TP=2, DFlash2 block-diffusion spec-decode

Quality

Serving

SM90 — H100 / H200 (recommended: TP=4)

SM120 — RTX PRO 6000 (Blackwell)

SM121 — 2× DGX Spark GB10 (desktop, 1M context)

Docker images

Quick start

Details

Validation

Research-stack known issue: DFlash2 admission wedge — root-caused across 21 legs; FIX VALIDATED (2026-09-04)

SM121 kernel-level research findings (2026-09-09)

License

4× H200 (SM90) — TP=4, published recipe, isl/osl 8192/1024, thinking ON (measured 2026-09-08, AWS p5en.48xlarge, image `glm53-flash-x86_64-cu130`)

4× H200 (SM90) — TP=4, published recipe, isl/osl 8192/1024, thinking ON (measured 2026-09-08, AWS p5en.48xlarge, image `glm53-flash-x86_64-cu130`)