brandonmusic

GLM-5.2-EXL3-DENSE6-MTP78

Quant recipe

Base: GLM-5.2, via the 3.0-bpw TR3 parent and the MTP-78 lineage recorded in the checkpoint configuration as malaiwah/GLM-5.2-EXL3-TR3-MTP78/3bpw-keep0.
Routed experts: TR3 3.0 bpw rank-sliced (hybrid_tr3_tail.moe_layers=[3,78]), byte-copied from the parent checkpoint.
Dense path: 375 tensors moved BF16 → EXL3 6.0 bpw — 75 each of o_proj, q_b_proj, shared_experts.{gate,up,down}_proj (layers 3–77). Per-tensor encode NMSE 3.8–5.1e-04.
Held in BF16 deliberately: lm_head, embed_tokens, router mlp.gate, DSA indexer, kv_b_proj, q_a_proj+kv_a_proj_with_mqa (packed fused_qkv_a_proj, n=576 fails Hadamard-128), all 15 dense tensors of layers 0–2, and the non-routed components of MTP layer 78. Layer-78 routed experts remain inherited EXL3 weights; the whole MTP layer is not BF16.
Why 6.0 bpw and not FP8/MXFP8 for the dense path: measured on real production weights, EXL3-6's output error floor (~2.0e-2 rel. Frobenius) is below fp8 per-out-channel (2.64e-2) and MXFP8 block=32 (2.39e-2) — e4m3's 3 mantissa bits are the binding constraint, not scale granularity. An fp8-dense experiment measured KLD 9.505 (vs 0.114) and was rejected.

Serving stack

Reference image (recommended): voipmonitor/vllm:gilded-gnosis-v20-vllm69ba80b-sia2ea608-fi801d57a-cu132-20260730-r13 (registry digest sha256:02796036c96a52fda0919aa260c45c70bc97d8e662a6ae5e614b5f987c20851b). Also validated on the 20260729 r11 image (…-vllm9502cc7-side7739a-…-r11).

Stock images do NOT serve this model as-is — two small patches are required (see patches/, applied via Dockerfile.r13pf):

patch_exl3_num_active.py — required or every TP worker dies at load with TypeError: exl3_moe(): incompatible function arguments. The vendored exllamav3 extension takes 30 args (num_active appended); vLLM's call site passes 29. Appends -1 (0 would silently zero the routed-expert output). Present in r11, r12 and r13 unchanged; until it lands upstream, this needs either the patch or a ~1-line vLLM PR. The checkpoint format itself (mixed 3.0/6.0 bpw, rank-sliced hybrid tail) loads natively — the loader resolves codebooks per layer and Exl3LinearMethod never references a global bit width — so no "new model type" PR is needed; only this call-site arity fix.
patch_exl3_prefetch.py — restores exllamav3's >144-row reconstruct dispatch for the dense path (materialize once + one cuBLAS HGEMM instead of re-decoding the trellis per 16-row m-tile). Without it, dense-quantized prefill loses ~25–37 %. Measured 4.76× on o_proj at M=2048; numerically verified vs the fused path (≤3.5e-03 worst rel. deviation). The file also contains an env-gated (default-off, falsified-on-measurement) aux-stream prefetch experiment — leave VLLM_EXL3_PREFETCH=0.

Build: cd patches && docker build -f Dockerfile.r13pf -t glm52-r13-pf:v1 . Then ./serve.sh (edit volume paths in the compose first).

Config and env as benchmarked (verified from `/proc/<pid>/environ`)

text

KV_CACHE_DTYPE=nvfp4_ds_mla   KV_FP8_ROPE=1   VLLM_NVFP4_MLA_DYNAMIC_SCALE=1VLLM_NVFP4_MLA_SCALES_FILE=   (explicitly empty → dynamic scale selection)MOE_BACKEND=b12x  MOE_MODE=a16  DCP=4  DCP_KV_CACHE_INTERLEAVE_SIZE=1VLLM_EXL3_RECONSTRUCT_M=144   VLLM_EXL3_PREFETCH=0--attention-backend B12X_MLA_SPARSE   --quantization exl3--max-num-seqs 8   --max-num-batched-tokens 3072cudagraph_capture_sizes=[4..32]   MTP-3 speculative decodingGPU_MEMORY_UTILIZATION=0.95 (launcher default)

Traps written in blood:

nvfp4_ds_mla survives only under B12X_MLA_SPARSE; anything else silently coerces to fp8_ds_mla. Dynamic NVFP4 scaling requires KV_FP8_ROPE=1.
The launcher chain defaults KV_CACHE_DTYPE to fp8 — pin it as a literal in the compose and verify from /proc/<pid>/environ, never from docker inspect (the launcher hard-exports values that don't appear there).
The first boot on a fresh compile cache JIT-compiles kernels during serving (r13 resolves its fused-MoE launch per forward with a live-batch cache key). Mount a persistent cache volume and treat the first session as warmup — never benchmark it. (One cold arm in our campaign read −25 % prefill purely from this.)

Hardware

4× NVIDIA RTX PRO 6000 Blackwell Workstation (2× Max-Q at a 300 W hard cap), sm_120a, 188 SMs, 96 GB GDDR7 each, PCIe Gen5, no NVLink. TP4 + DCP4.

KV-cache capacity (as served, GMU 0.95)

Table with columns: tokens, note
	tokens	note
nvfp4_ds_mla KV (this config)	1,619,968 (r13) / 1,618,432 (r11)	3.09× concurrency at 524,288-token max-len
nvfp4 + capacity patches (`Dockerfile.r13cap`)	1,758,720 (+8.6%)	3.35× — shares the 1054 MiB/rank draft prefill arena; speed-neutral (paired A/B), KLD unchanged, estonia 5/5
fp8 KV baseline (same weights)	~1,126,400	nvfp4 = +43.7 %

Quality

Table with columns: gate, target, measured
gate	target	measured
MTP accept α	≥ 2.828	3.245 (n=3: 3.31/3.24/3.19)
KLD vs BF16-dense parent (nvfp4-dyn KV)	≤ 0.112645 + noise	0.114391 ± 0.00192 (n=5, r11) / 0.113777 ± 0.00212 (n=5, r13) — 1.3σ
KLD with fp8 KV (same weights)	—	0.105479 ± 0.00139 (n=5) — lower-KLD / lower-capacity operating point
estonia (×20, concurrency 5, temp 0)	—	20/20
lavd ledger (×20, concurrency 5, temp 0)	—	19/20 (12 exact, 7 near, 1 fail)
hotel-lights (×20, c5, temp 0, rp 1.15, 64k cap)	—	12/20 exact
hotel-lights (×20, c5, temp 0, no rp, uncapped)	—	7/20 — flag-confounded: greedy with no repetition penalty loops; one run generated ~3 h before abort. Use a rep penalty with this profile.
hotel-lights (×15, c5, reduced thinking, rp 1.15, 20k cap)	—	2/15 — reduced reasoning effort collapses this suite (and/or 20k truncation); serve reasoning-heavy workloads at default (Max) effort
needle-in-haystack, 32k→350k × depths 10/25/50/75/90	—	35/35 recovered, zero garbled — perfect retrieval at every context (32k, 64k, 128k, 192k, 256k, 320k, 350k) and every depth
fused-GEMM numerics r11 vs r12 vs r13	—	bit-identical (probe on real 6 bpw weight, 1.687e-03 vs fp32-accumulate reference)

KLD harness: 2048-token teacher-forced eval vs BF16 reference logits, n=5 runs, TP4 eager. Reference arms on the parent TR3 checkpoint: uncalibrated 0.151127 / static scales 0.116195 / dynamic 0.112645 / fp8-KV 0.100246.

Speed (position-matched, cold-start-gated, n=3 per arm unless noted)

Prefill = standalone, server-side tok/s. Decode = C1 per-request (1/ITL), MTP-3, temp 0.

Table with columns: arm, prefill 8k, prefill 32k, decode C1
arm	prefill 8k	prefill 32k	decode C1
r13 + patches, DENSE6 (this repo)	2224 ± 16	2090 ± 5	101.2 ± 0.8
r11 + patches, DENSE6	2234 ± 26	2095 ± 28	93.2 ± 3.7
r11 + patches, parent TR3 (BF16 dense)	2330 ± 1	2185 ± 2	97.9 ± 1.0

The dense quant costs −4.1 % prefill / −4.7 % decode vs the BF16-dense parent on the same image — the price of +43.7 % KV, −15 GiB disk, and the KLD floor above.
r13 vs r11 on this checkpoint: prefill parity, decode +9.0 % (exact-m (128,128) decode-window kernels), KLD unchanged, kernel numerics bit-identical.

Full sweep on r13 (single round, results/battery/):

Table with columns: context, prefill tok/s, concurrency, decode aggregate (ctx0 / 32k)
context	prefill tok/s	concurrency	decode aggregate (ctx0 / 32k)
8k	2261	c1	100 / 100
16k	1403 †	c2	153 / 149
32k	2108	c4	225 / 219
64k	1993	c8	309 / 284
128k	1813

† n=1 anomaly, off-trend TTFT; consistent with a one-time JIT compile on a context-length kernel bucket (see traps above), re-measure pending.

Measurement discipline (why the error bars exist)

Prefill on this rig is thermally ordered, not noisy — arms alternate cold-started (<40 °C gate) in Latin-square position rotation. n=1 is never evidence; three separate campaign conclusions were reversed at n=3, and a fourth (a phantom "r13 regression") was a first-boot compile artifact caught by position rotation. Full campaign narrative and negative results (fp8/MXFP8 dense, k58 GEMV, fp16 MMA accumulation, aux-stream reconstruct prefetch — falsified at 0.3 % of end-to-end) are in the source project's REPORT.

Repo layout

text

launcher/                           the ACTUAL serving entrypoint chain from the                                    image (serve-gilded-gnosis.sh -> v19 -> v16):                                    every env var above is consumed herecompose/dense6-r13pf/compose.yml   serving compose (r13, recommended)compose/dense6-r11pf/compose.yml   serving compose (r11, also validated)patches/                            the two required patches + Dockerfilesserve.sh                            bring-up + /proc env verificationresults/battery/                    full bench + estonia/lavd/hotel/needle (JSON+logs)results/tri-campaign/               the 10 position-matched speed armsresults/kld/                        KLD runs: r13 nvfp4-dyn, fp8-KV frontier, r11 canonicalresults/sharegate/                  in-situ reconstruct share measurement (0.3 %)

Credits and lineage

Z.ai for GLM-5.2.
Brandon Music for the EXL3/TR3 encoder work, owner corpus, DENSE6 derivative construction, validation, and publication.
turboderp for ExLlamaV3 v0.0.43 and EXL3.
malaiwah for the MTP-78 overlay and calibration capture.
Josh Cartu for the MTP-78 recipe and rank-sliced runtime work.
Luke Alonso for B12X and the associated Blackwell kernel/runtime work.
Martin Vit and yatesdr for the Infernal Invocation / RC2 image engineering credited by the upstream MTP-78 overlay.
Special thanks to local-inference-lab for its vLLM fork, image, review, testing, and release engineering.

Canonical project attribution is also recorded in the TR3 quantization provenance.

License

Inherits the license of the base GLM-5.2 model — see the base model's terms before use. The patches and scripts in this repo are provided as-is.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

brandonmusic

Model Tree

Base

brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw

Quantized

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

Quant recipe

Base: GLM-5.2, via the 3.0-bpw TR3 parent and the MTP-78 lineage recorded in the checkpoint configuration as malaiwah/GLM-5.2-EXL3-TR3-MTP78/3bpw-keep0.
Routed experts: TR3 3.0 bpw rank-sliced (hybrid_tr3_tail.moe_layers=[3,78]), byte-copied from the parent checkpoint.
Dense path: 375 tensors moved BF16 → EXL3 6.0 bpw — 75 each of o_proj, q_b_proj, shared_experts.{gate,up,down}_proj (layers 3–77). Per-tensor encode NMSE 3.8–5.1e-04.
Held in BF16 deliberately: lm_head, embed_tokens, router mlp.gate, DSA indexer, kv_b_proj, q_a_proj+kv_a_proj_with_mqa (packed fused_qkv_a_proj, n=576 fails Hadamard-128), all 15 dense tensors of layers 0–2, and the non-routed components of MTP layer 78. Layer-78 routed experts remain inherited EXL3 weights; the whole MTP layer is not BF16.
Why 6.0 bpw and not FP8/MXFP8 for the dense path: measured on real production weights, EXL3-6's output error floor (~2.0e-2 rel. Frobenius) is below fp8 per-out-channel (2.64e-2) and MXFP8 block=32 (2.39e-2) — e4m3's 3 mantissa bits are the binding constraint, not scale granularity. An fp8-dense experiment measured KLD 9.505 (vs 0.114) and was rejected.

Serving stack

Stock images do NOT serve this model as-is — two small patches are required (see patches/, applied via Dockerfile.r13pf):

patch_exl3_num_active.py — required or every TP worker dies at load with TypeError: exl3_moe(): incompatible function arguments. The vendored exllamav3 extension takes 30 args (num_active appended); vLLM's call site passes 29. Appends -1 (0 would silently zero the routed-expert output). Present in r11, r12 and r13 unchanged; until it lands upstream, this needs either the patch or a ~1-line vLLM PR. The checkpoint format itself (mixed 3.0/6.0 bpw, rank-sliced hybrid tail) loads natively — the loader resolves codebooks per layer and Exl3LinearMethod never references a global bit width — so no "new model type" PR is needed; only this call-site arity fix.
patch_exl3_prefetch.py — restores exllamav3's >144-row reconstruct dispatch for the dense path (materialize once + one cuBLAS HGEMM instead of re-decoding the trellis per 16-row m-tile). Without it, dense-quantized prefill loses ~25–37 %. Measured 4.76× on o_proj at M=2048; numerically verified vs the fused path (≤3.5e-03 worst rel. deviation). The file also contains an env-gated (default-off, falsified-on-measurement) aux-stream prefetch experiment — leave VLLM_EXL3_PREFETCH=0.

Build: cd patches && docker build -f Dockerfile.r13pf -t glm52-r13-pf:v1 . Then ./serve.sh (edit volume paths in the compose first).

Config and env as benchmarked (verified from `/proc/<pid>/environ`)

text

KV_CACHE_DTYPE=nvfp4_ds_mla   KV_FP8_ROPE=1   VLLM_NVFP4_MLA_DYNAMIC_SCALE=1VLLM_NVFP4_MLA_SCALES_FILE=   (explicitly empty → dynamic scale selection)MOE_BACKEND=b12x  MOE_MODE=a16  DCP=4  DCP_KV_CACHE_INTERLEAVE_SIZE=1VLLM_EXL3_RECONSTRUCT_M=144   VLLM_EXL3_PREFETCH=0--attention-backend B12X_MLA_SPARSE   --quantization exl3--max-num-seqs 8   --max-num-batched-tokens 3072cudagraph_capture_sizes=[4..32]   MTP-3 speculative decodingGPU_MEMORY_UTILIZATION=0.95 (launcher default)

KV_CACHE_DTYPE=nvfp4_ds_mla   KV_FP8_ROPE=1   VLLM_NVFP4_MLA_DYNAMIC_SCALE=1VLLM_NVFP4_MLA_SCALES_FILE=   (explicitly empty → dynamic scale selection)MOE_BACKEND=b12x  MOE_MODE=a16  DCP=4  DCP_KV_CACHE_INTERLEAVE_SIZE=1VLLM_EXL3_RECONSTRUCT_M=144   VLLM_EXL3_PREFETCH=0--attention-backend B12X_MLA_SPARSE   --quantization exl3--max-num-seqs 8   --max-num-batched-tokens 3072cudagraph_capture_sizes=[4..32]   MTP-3 speculative decodingGPU_MEMORY_UTILIZATION=0.95 (launcher default)

Traps written in blood:

nvfp4_ds_mla survives only under B12X_MLA_SPARSE; anything else silently coerces to fp8_ds_mla. Dynamic NVFP4 scaling requires KV_FP8_ROPE=1.
The launcher chain defaults KV_CACHE_DTYPE to fp8 — pin it as a literal in the compose and verify from /proc/<pid>/environ, never from docker inspect (the launcher hard-exports values that don't appear there).
The first boot on a fresh compile cache JIT-compiles kernels during serving (r13 resolves its fused-MoE launch per forward with a live-batch cache key). Mount a persistent cache volume and treat the first session as warmup — never benchmark it. (One cold arm in our campaign read −25 % prefill purely from this.)

Hardware

4× NVIDIA RTX PRO 6000 Blackwell Workstation (2× Max-Q at a 300 W hard cap), sm_120a, 188 SMs, 96 GB GDDR7 each, PCIe Gen5, no NVLink. TP4 + DCP4.

KV-cache capacity (as served, GMU 0.95)

Table with columns: tokens, note
	tokens	note
nvfp4_ds_mla KV (this config)	1,619,968 (r13) / 1,618,432 (r11)	3.09× concurrency at 524,288-token max-len
nvfp4 + capacity patches (`Dockerfile.r13cap`)	1,758,720 (+8.6%)	3.35× — shares the 1054 MiB/rank draft prefill arena; speed-neutral (paired A/B), KLD unchanged, estonia 5/5
fp8 KV baseline (same weights)	~1,126,400	nvfp4 = +43.7 %

Quality

Table with columns: gate, target, measured
gate	target	measured
MTP accept α	≥ 2.828	3.245 (n=3: 3.31/3.24/3.19)
KLD vs BF16-dense parent (nvfp4-dyn KV)	≤ 0.112645 + noise	0.114391 ± 0.00192 (n=5, r11) / 0.113777 ± 0.00212 (n=5, r13) — 1.3σ
KLD with fp8 KV (same weights)	—	0.105479 ± 0.00139 (n=5) — lower-KLD / lower-capacity operating point
estonia (×20, concurrency 5, temp 0)	—	20/20
lavd ledger (×20, concurrency 5, temp 0)	—	19/20 (12 exact, 7 near, 1 fail)
hotel-lights (×20, c5, temp 0, rp 1.15, 64k cap)	—	12/20 exact
hotel-lights (×20, c5, temp 0, no rp, uncapped)	—	7/20 — flag-confounded: greedy with no repetition penalty loops; one run generated ~3 h before abort. Use a rep penalty with this profile.
hotel-lights (×15, c5, reduced thinking, rp 1.15, 20k cap)	—	2/15 — reduced reasoning effort collapses this suite (and/or 20k truncation); serve reasoning-heavy workloads at default (Max) effort
needle-in-haystack, 32k→350k × depths 10/25/50/75/90	—	35/35 recovered, zero garbled — perfect retrieval at every context (32k, 64k, 128k, 192k, 256k, 320k, 350k) and every depth
fused-GEMM numerics r11 vs r12 vs r13	—	bit-identical (probe on real 6 bpw weight, 1.687e-03 vs fp32-accumulate reference)

Speed (position-matched, cold-start-gated, n=3 per arm unless noted)

Prefill = standalone, server-side tok/s. Decode = C1 per-request (1/ITL), MTP-3, temp 0.

Table with columns: arm, prefill 8k, prefill 32k, decode C1
arm	prefill 8k	prefill 32k	decode C1
r13 + patches, DENSE6 (this repo)	2224 ± 16	2090 ± 5	101.2 ± 0.8
r11 + patches, DENSE6	2234 ± 26	2095 ± 28	93.2 ± 3.7
r11 + patches, parent TR3 (BF16 dense)	2330 ± 1	2185 ± 2	97.9 ± 1.0

The dense quant costs −4.1 % prefill / −4.7 % decode vs the BF16-dense parent on the same image — the price of +43.7 % KV, −15 GiB disk, and the KLD floor above.
r13 vs r11 on this checkpoint: prefill parity, decode +9.0 % (exact-m (128,128) decode-window kernels), KLD unchanged, kernel numerics bit-identical.

Full sweep on r13 (single round, results/battery/):

Table with columns: context, prefill tok/s, concurrency, decode aggregate (ctx0 / 32k)
context	prefill tok/s	concurrency	decode aggregate (ctx0 / 32k)
8k	2261	c1	100 / 100
16k	1403 †	c2	153 / 149
32k	2108	c4	225 / 219
64k	1993	c8	309 / 284
128k	1813

† n=1 anomaly, off-trend TTFT; consistent with a one-time JIT compile on a context-length kernel bucket (see traps above), re-measure pending.

Measurement discipline (why the error bars exist)

Repo layout

text

launcher/                           the ACTUAL serving entrypoint chain from the                                    image (serve-gilded-gnosis.sh -> v19 -> v16):                                    every env var above is consumed herecompose/dense6-r13pf/compose.yml   serving compose (r13, recommended)compose/dense6-r11pf/compose.yml   serving compose (r11, also validated)patches/                            the two required patches + Dockerfilesserve.sh                            bring-up + /proc env verificationresults/battery/                    full bench + estonia/lavd/hotel/needle (JSON+logs)results/tri-campaign/               the 10 position-matched speed armsresults/kld/                        KLD runs: r13 nvfp4-dyn, fp8-KV frontier, r11 canonicalresults/sharegate/                  in-situ reconstruct share measurement (0.3 %)

launcher/                           the ACTUAL serving entrypoint chain from the                                    image (serve-gilded-gnosis.sh -> v19 -> v16):                                    every env var above is consumed herecompose/dense6-r13pf/compose.yml   serving compose (r13, recommended)compose/dense6-r11pf/compose.yml   serving compose (r11, also validated)patches/                            the two required patches + Dockerfilesserve.sh                            bring-up + /proc env verificationresults/battery/                    full bench + estonia/lavd/hotel/needle (JSON+logs)results/tri-campaign/               the 10 position-matched speed armsresults/kld/                        KLD runs: r13 nvfp4-dyn, fp8-KV frontier, r11 canonicalresults/sharegate/                  in-situ reconstruct share measurement (0.3 %)

Credits and lineage

Z.ai for GLM-5.2.
Brandon Music for the EXL3/TR3 encoder work, owner corpus, DENSE6 derivative construction, validation, and publication.
turboderp for ExLlamaV3 v0.0.43 and EXL3.
malaiwah for the MTP-78 overlay and calibration capture.
Josh Cartu for the MTP-78 recipe and rank-sliced runtime work.
Luke Alonso for B12X and the associated Blackwell kernel/runtime work.
Martin Vit and yatesdr for the Infernal Invocation / RC2 image engineering credited by the upstream MTP-78 overlay.
Special thanks to local-inference-lab for its vLLM fork, image, review, testing, and release engineering.

Canonical project attribution is also recorded in the TR3 quantization provenance.

License

Inherits the license of the base GLM-5.2 model — see the base model's terms before use. The patches and scripts in this repo are provided as-is.

GLM-5.2-EXL3-DENSE6-MTP78

README

Quant recipe

Serving stack

Config and env as benchmarked (verified from `/proc/<pid>/environ`)

Hardware

KV-cache capacity (as served, GMU 0.95)

Quality

Speed (position-matched, cold-start-gated, n=3 per arm unless noted)

Measurement discipline (why the error bars exist)

Repo layout

Credits and lineage

License

Explore FriendliAI today

README

Quant recipe

Serving stack

Config and env as benchmarked (verified from `/proc/<pid>/environ`)

Hardware

KV-cache capacity (as served, GMU 0.95)

Quality

Speed (position-matched, cold-start-gated, n=3 per arm unless noted)

Measurement discipline (why the error bars exist)

Repo layout

Credits and lineage

License

GLM-5.2-EXL3-DENSE6-MTP78

README

Quant recipe

Serving stack

Config and env as benchmarked (verified from /proc/<pid>/environ)

Hardware

KV-cache capacity (as served, GMU 0.95)

Quality

Speed (position-matched, cold-start-gated, n=3 per arm unless noted)

Measurement discipline (why the error bars exist)

Repo layout

Credits and lineage

License

Explore FriendliAI today

README

Quant recipe

Serving stack

Config and env as benchmarked (verified from /proc/<pid>/environ)

Hardware

KV-cache capacity (as served, GMU 0.95)

Quality

Speed (position-matched, cold-start-gated, n=3 per arm unless noted)

Measurement discipline (why the error bars exist)

Repo layout

Credits and lineage

License

Config and env as benchmarked (verified from `/proc/<pid>/environ`)

Config and env as benchmarked (verified from `/proc/<pid>/environ`)