davidsyoung

GLM-5.3-EXL3-TR3-3.42bpw

README

License: glm-5.3

What is quantized

Table with columns: component, treatment
component	treatment
routed experts (layers 3–78, incl. MTP-78)	EXL3 trellis, per layer 148 experts K3 + 108 experts K4 (avg 3.42 bpw), mcg codebook
dense MLP (layers 0–2), all attention, norms, embeddings, lm_head, mlp.gate, eh_proj	BF16, carried byte-exact
shared experts	BF16 in-checkpoint (online K6 at serve)

Data-free encode: identity Hessian (H=I), rotations + trellis search only — no calibration capture anywhere. Deterministic: seeds derived from (layer, expert, projection, rank), seed_base 20260711.
K4 selection: per layer, the experts with highest relative round-trip MSE under the K3 encode are re-encoded at K4 (worst-108 per layer: 64 from the 3.25 pass + next-44 delta pass, identical seeds/machinery).
Per-expert tier map in tier_bitmap.json; encode provenance in config.json.hybrid_tr3_tail; file hashes in MANIFEST.sha256.

Serving

TR3 mixed-bit layout (carrier BF16 shards + per-layer trellis payloads) — serve with an exllamav3-b12x/sparkinfer-lineage stack, TP4 (+ DCP4/MTP3), FP8 KV. The mixed-K projection-tiers patch is REQUIRED; a stock loader that assumes a uniform K per layer will produce fluent garbage. Not loadable by vanilla exllamav3 model loading.

Independent turnkey serving qualification (2026-08-29)

A separate live qualification on four RTX PRO 6000 Blackwell Server Edition GPUs (97,887 MiB each; NVIDIA 595.58.03 / CUDA 13.2) passed with the fail-closed turnkey image and public profile below:

Image: ghcr.io/malaiwah/glm52-exl3-vast@sha256:aec4075e02241b71321b5601763eba35e86ea02b13c3c3c43ac34fae30161160
Profile: MODEL_PROFILE=glm53-3.42bpw
Checkpoint revision: 8bef807a0fcdd180e984a26b50e731cdba9a8ff2
Topology: TP4 + DCP4, online shared-expert K6, native probabilistic MTP3, mixed K3/K4 routed experts, dynamic-token NVFP4 KV with FP8 RoPE, C8
Qualified request envelope: 393,216 tokens and 3,415,867,392 KV-cache bytes per GPU. This is a measured safe profile, not the architecture's maximum.

The inherited 520,192-token GLM-5.2 envelope passed deterministic startup checks but OOMed during the first temperature-1 sampler compile; it is not a qualified setting for this checkpoint. The selected arm retained at least 665 MiB of observed physical free memory after first-use compilation and C8 sampling.

Observed gates: startup arithmetic/factual/instruction plus strict JSON and 32K retrieval; all applicable OpenAI chat/streaming/reasoning/structured-output/ tool-use contracts; 72/72 temperature-1 decode requests across C1/C4/C8 (227.55 output tok/s at C8); and 15/15 five-depth retrieval facts through a 389,959-token tokenizer-exact document. The post-stress verifier and 3,469-line runtime-log audit were clean. A second boot from the exact published image passed authentication (401 without a key, 200 with a key), chat inference, and the token-gated dashboard while running with no source-code bind mounts.

Full measurements, rejected-arm evidence, and artifact names: qualification record. The image carries the required mixed-K projection-tier runtime rather than silently loading this checkpoint with a uniform-K path.

KLD vs BF16 teacher

Full-vocabulary, teacher-forced KL(teacher || student) against sealed BF16 GLM-5.3 logits — 4 held-out windows × 2,047 positions × 154,880 vocabulary, fp32 log-softmax both sides. Measured independently on two different 4× RTX PRO 6000 (96GB) machines:

Table with columns: weight quant, KV mode, this work, CN3 (@dareposte), Δ
weight quant	KV mode	this work	CN3 (@dareposte)	Δ
3.42 bpw	fp8	0.024105	0.023966	−0.6%
3.25 bpw	fp8	0.026103	0.026776	+2.6%
3.25 bpw	nvfp4

Readings: the 3.25↔3.42 weight step changes KLD by only ~0.002; the fp8→nvfp4 KV step costs ~7× more (~0.014) — cache format matters more than the extra 0.17 bpw. Under nvfp4 KV the weight-quant difference washes out entirely. The window SD (~0.02–0.03) is corpus heterogeneity — one citation-dense legal window is uniformly hardest; dialogue, explanatory prose, and reasoning-trace registers measure near-transparent.

Table with columns: config, source, w0000, w0001, w0002, w0003
config	source	w0000	w0001	w0002	w0003
3.42 fp8	this work	0.0141	0.0537	0.0137	0.0148
3.42 fp8	CN3	0.0148	0.0542	0.0135	0.0134
3.25 fp8	this work	0.0188

Method (reproducible)

Teacher: brandonmusic/GLM-5.3-BF16-full-logits, reference-full-panel confirmation lane (held out from every calibration fit), revision 427368f1.
Student: this checkpoint, loaded by the digest-pinned r17 serving image (sha256:c5e96c5b…) — the real trellis kernels and online-K6 path, offline vllm.LLM, TP4, one teacher-forced prefill per window.
Full runbook + runner: kld/ in this repo (KLD-REPRODUCTION.md, prefill_kld_53.py, fetch-teacher.sh).
Independent reproduction bundle (receipts, unedited logs, checksums, pinned revisions): .

Capacity caveat (via the CN3 report): KV-pool token figures printed by KLD-profile boots (TP4/DCP1, 4,096-token envelope) are logical pool values for that profile only — they are not maximum context length or production serving capacity.

Credits

brandonmusic — thank you for the GLM-5.3-BF16-full-logits teacher captures that make this measurement possible without a 1.5TB BF16 forward, and for the TR3 quantization references this release follows: the GLM-5.2-EXL3-TR3v4-3.5bpw-MTP78 method card, runbook, and the r10 reproducibility bundle whose encoder lineage (encode_tr3_v31.py) this checkpoint was produced with.
dareposte — thank you for the independent CN3 reproduction: all six weight/KV configurations on separate hardware, within ±5% of our means, published here with receipts under kld/cn3/.
willfalco/GLM-5.2-EXL3-TR3-3.25bpw — the 3.25 tier recipe and mixed-K checkpoint-format lineage.

Status

Measured and independently reproduced 2026-08-29: mean KLD 0.024105 (fp8 KV; CN3 reproduction 0.023966) — the quality artifact of this release pair. Per-layer K4 set is a strict superset of the 3.25 release.

Quantized with encode_tr3_53.py (exllamav3 v0.0.43 vendored math, MIT) on 4x RTX PRO 6000 Blackwell. Encode + tooling notes ship in the repo.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

davidsyoung

Model Tree

Base

zai-org/GLM-5.3

Quantized

this model

Input Modalities

Text

Output Modalities