jpsequeira

GLM-5.2-EXL3-TR3-3.40bpw-KVarN-K4V2

What is different here

KVarN K4V2 KV cache — 290 B/token/layer latent-tile records with a self-describing dtype (kvarn_mla_k2_g64). Smaller than compact NVFP4 MLA KV (368 B) and roughly 2.3x smaller than FP8 (656 B), at higher measured fidelity than both on this model.
b12x (B12X_MLA_SPARSE) backend — sparse MLA attention with the paged top-k indexer, CKV gather prefill, and direct-packed DCP4 decode; deterministic top-k merge at zero measured cost.
local-inference-lab/vllm fork — the runtime base this stack is developed against; the serving image bakes the fork plus b12x, KVarN, and sparkinfer kernels into one bootable unit.

Checkpoint design: the MCG promotion ladder

The TR3 allocation was built in one evidence-driven pass, not uniform bits:

Full MCG3 base quantization — routed experts quantized to MCG3 using captures from the BF16 model on our own corpus.
Activation-aware promotion — candidates for promotion were selected by measured activation alignment on coding, reasoning, and agentic workloads (the domains this checkpoint is built for).
MCG4 as the promotion ceiling — aligned candidates were promoted MCG3 -> MCG4; nothing went higher because no measurement justified it.
Width milestones 3.00 -> 3.25 -> 3.38 -> 3.40 bpw — each step was validated, and 3.40 was the last balance point: pushing further would have forced cutting context capacity below the serving contract.
KVarN was a from-the-start design focus, not a retrofit — the KV format's size/quality potential shaped the whole budget, and the measured payoff (290 B/token at KLD 0.0558) is what makes 512K context coexist with the lowest published KLD.

Positioning vs the community checkpoints

brandonmusic's 3.5 bpw R7 optimizes for general-quality evidence with a broader (including legal-domain) emphasis. This checkpoint deliberately skews the other way: the promotion ladder weighted coding, reasoning, and agentic activation evidence, and the width stopped at 3.40 bpw to protect context capacity. willfalco's 3.42 bpw is the closest sibling in spirit (coding/agentic first); the difference here is KVarN KV at 290 B/token — smaller and more faithful than the KV formats in either recipe — which is what buys the 512K context alongside the quality.

Correction & retraction: KLD claim (2026-08-19)

We retract the claim that this checkpoint has "the lowest directly-measured KLD published for GLM-5.2 EXL3 at any width." That claim was measured on our calibration-domain panel (item13, the same corpus family the quantization allocation was tuned on). On held-out data with the community methodology, the claim does not hold. Credit to brandonmusic and willfalco, whose public held-out measurements and reference logits (festr2/GLM-5.2-BF16-KLD-Reference-Logits-20260708) prompted and enabled this correction.

Held-out measurement (wikitext-2 test window, 2,048 tokens, festr2 BF16 reference logits, fp8-KV reference per their methodology, same 3.40 bpw checkpoint varying only KV format):

Table with columns: KV cache, forward KLD (mean ± sd), top-1 agreement
KV cache	forward KLD (mean ± sd)	top-1 agreement
Standard FP8	0.0871 ± 0.0012	94.4%
KVarN K4V2 (this model's default)	0.0889 ± 0.0016	94.4%
KVarN K5	0.0900 ± 0.0012 (n=3)	94.3%
Dynamic NVFP4	0.1213 ± 0.0022	92.8%

For comparison on the same held-out reference (matched fp8-KV configs): brandonmusic's 3.5 bpw TR3v4 measures 0.0654 ± 0.0021, and willfalco's 3.36 bpw EXL3 measures 0.0685 — both closer to BF16 than our 3.40 bpw checkpoint. Since a narrower quant (3.36) outperforms ours, the gap is attributable to allocation, not width: our promotion scheme (activation-weighted on a coding/reasoning/agentic calibration corpus) appears overfit to that corpus — 0.0555 on our calibration panel vs 0.0871 held-out at identical config. The calibration-domain numbers remain correct measurements of in-distribution fidelity; they are not comparable to held-out numbers and should not be quoted as general quality.

What still stands:

KV economics: K4V2 costs +0.0016 over FP8; the community's Dynamic NVFP4 costs +0.0344 on the same weights — 20x the format charge. At 290 B/token vs 368, this is a smaller cache that is also more faithful.
Context: 819K proven (vs ~347K for nvfp4-KV deployments at this hardware class).
The item13 panel, plan SHA 072e37f..., and per-row token_ids_sha256 values are published for full reproduction of our calibration-domain numbers.

We intend to re-measure and, if a re-allocation campaign succeeds, re-enter the held-out comparison on these terms.

Quality evidence (calibration-domain — in-distribution; see the retraction above before quoting these numbers)

Direct BF16-reference KLD, full corpus item13 captures, 2,047 positions:

Table with columns: Configuration, KLD vs BF16
Configuration	KLD vs BF16
TR3 3.40 bpw + KVarN K4V2 KV	0.0558
TR3 3.40 bpw + KVarN K4 KV (418 B)	0.0558
TR3 3.40 bpw + KVarN K5 KV (482 B)	0.0554

For context only (different evaluators and procedures — not a leaderboard): brandonmusic 3.5 bpw R7 reported 0.0613 +/- 0.0014; the standardized 3.42 bpw matrix reported 0.0741 with FP8 KV and 0.1080 with NVFP4 MLA KV. The KVarN numbers above were measured with the procedure in -assets/kld/ (capture plan, reference, and reports are archived for reproduction).

Long-context: estonia-long 29/30 answers at 133K-token prompts.

Measured serving profile

TP4/DCP4/MTP3, one PCIe switch, 290 W locks, greedy:

Table with columns: Metric, Value
Metric	Value
Decode C1	86-91 tok/s (wall 32-34 ms, AL 2.9)
Decode c2 / c4 (256K pool)	104 / 168 tok/s
Prefill 8K-64K	2.4-2.5k tok/s
TTFT 32K	~13.5 s
Context	512K native (768K bootable)

KV capacity field observations: 512K at 2000 blocks/GPU with the c1-focused profile; the c4-capable 256K pool trades context for the c1/c2/c4 = 86/104/168 tok/s grid. DCP2 at 360K context is enabled and validated at champion speeds (decode 89, prefill 2,433); DCP1 is unsupported for KVarN and fails closed at config time.

Quick start

Runtime image (public): jmpsequeira/glm52-kvarn-k4v2:tr3 (all code baked — b12x, KVarN reader, sparkinfer, EXL3 loader).

IMPORTANT — re-pull the image if you downloaded it before 2026-08-19. Earlier pulls contain two prefix-caching bugs (KVarN KV + --enable-prefix-caching): garbage output on cache hits and persistent server-wide corruption after long requests. Fixed in tr3.2 (digest df889e27461a), which is also the current tr3. docker run does NOT auto-update a cached tag — run docker pull jmpsequeira/glm52-kvarn-k4v2:tr3 (delta download, seconds) or pin tr3.2. With the fix, prefix caching is safe and recommended; quality is unchanged (KLD 0.0555 vs 0.0558 of-record).

RESOLVED (2026-08-22) — Root cause found: not corruption. The 2-bit KV format (kvarn_mla_k2_g64) is below the generation-coherence floor. Every store/read kernel was verified mathematically sound (pack/rehydrate round-trip matches an independent reference exactly; the native SM120 decode reader agrees with record bytes to 0.3%; pool latents confirmed Hadamard-gaussianized; ownership/mapping verified valid at 55K probed decode steps). The 2-bit latent grid carries an inherent ~45% relative quantization error (measured; matches pure-RTN theory) — sufficient for prompt-side KL panels (short contexts are served from the FP8 exact pool and never touch packed rows — the 0.0558 KLD was an evaluation blind spot), but beyond what generation tolerates whenever quantized rows are attended: hot-window packing (PRECISION_TAIL_TOKENS=0) garbles immediately; old-row selection at larger tails garbles sporadically under history pressure. 4-bit (kvarn_mla_k4_g64, ~9% error) is clean in identical batteries when the hot window stays exact.

Layer-wise mixed precision — RETRACTED for serving (2026-08-23): the runtime supports demoting layers to the 2-bit record (KVARN_MLA_K2_LAYERS), and late layers (40-77) passed every standard gate (generation battery, tool calls, 30-turn soak, estonia-long 10/10 at 133K, cache-eviction churn). It is nonetheless too unstable for production: prefix-cache rehydration corrupts through the early layers' degenerate subnormal latent tiles, and extreme contexts (~700K) poison the serving boot outright. Until the format gains a story for those regimes, serve pure kvarn_mla_k4_g64 with KVARN_MLA_PRECISION_TAIL_TOKENS=3072.

k2-mix UPDATE (, 2026-08-26): the retraction's churn-corruption leg is disproven — it was the general records-ptr provenance wipe fixed in . Layers 40-77 demoted to on a k4 base pass the full churn soak post-fix (pressures, re-entries at full APC hit, exact needle retrieval). The k2-mix (~356 B/token, ~15% KV savings) is viable again for contexts ≤512K. The retraction now rests on a single remaining defect: ~700K contexts poison the serving boot (CKV lease issue, unfixed). The 2-bit record remains useful only for short-context, read-mostly workloads.

RESOLVED in tr3.4 (2026-08-26, digest f65645545938) — general cache-eviction corruption: the probabilistic, state-accumulating KV corruption under multi-100K eviction churn (all KV dtypes, worst under 2-bit mixes) was traced to a worker-side bookkeeping defect: a single shared records-pointer slot was alternated by per-layer rebinds and misread as record-storage reallocation, wiping flush provenance every rebind cycle and silently disabling record rehydrate on prefix-cache hits after eviction. Fixed by per-impl pointer keying; verified by a deterministic double-eviction reproducer and a 4-round churn soak (coherent re-entries at full 118K-token APC hit, exact needle retrieval). docker pull jmpsequeira/glm52-kvarn-k4v2:tr3.4 (also :latest). Earlier tags (tr3.3 and below) still carry the defect — the churn caveat applies to them only. This also re-attributes the 2-bit retraction's "rehydration corrupts through subnormal latent tiles" mechanism: that corruption was this bug; the 2-bit retraction stands on its remaining evidence (~700K boot poisoning, degraded gates).

tr3.5 (2026-08-26, digest 61ab86886a9c): fp8-RoPE record cell — opt-in. The KVarN packed record's RoPE rows move from BF16 to E4M3 + per-token fp16 amax (the #95 cell design): 418 → 356 B/token (−14.8%), ~15% more context per KV pool. Enable with KVARN_FP8_ROPE_RECORD=1. Validated end-to-end on the churn battery at 327680 ctx (pressures, 1s full-hit re-entries through fp8 records, exact needle retrieval, canary). Default OFF — tr3.5 without the flag is byte-identical serving behavior to tr3.4.

Serve with --kv-cache-dtype kvarn_mla_k4_g64 and KVARN_MLA_PRECISION_TAIL_TOKENS=3072 (the launcher defaults). The KVarN KLD table's 2-bit rows are valid only for short-context workloads. Separately filed: an MTP accepted-token-resolution race (fail-closed crash, not corruption), exposed under heavy per-step instrumentation — tracked in the runtime repo. root cause and regression suite once landed).

bash
docker run --rm --name glm52-kvarn \
  --gpus '"device=0,1,2,3"' --network host --ipc host \
  --ulimit memlock=-1:-1 --ulimit stack=67108864:67108864 \
  -v ~/.cache/huggingface:/hf-cache:ro \
  -v "$PWD/vllm-cache:/cache" \
  jmpsequeira/glm52-kvarn-k4v2:tr3 \
  --model jpsequeira/GLM-5.2-EXL3-TR3-3.40bpw-KVarN-K4V2 \
  ... (vllm serve flags per profile below)

The model is sourced through the Hugging Face cache: mount your ~/.cache/huggingface read-only and pass the repo id as --model (HF_HUB_CACHE=/hf-cache/hub, offline resolution — no download if the snapshot is cached, automatic fetch if not). The MTP head is layer 78, merged natively into this checkpoint — no draft dir, no spec "model" key; the loader sources model.layers.78.* from the main index like every other layer (EXL3 MCG3 3.0 bpw experts in projection-mixed-layer-078, shared-H rotations in shared-h-layer-078, BF16-protected modules in bf16-protected-layer-078). VLLM_EXL3_TRELLIS_MIN_M=1 is mandatory. Measured zero acceptance cost (AL 2.85/2.84 in-band vs the NVFP4 draft).

Core profile env/flags (the full validated contract ships in the -src repo's boot-from-image.sh):

TP=4, DCP=4, MTP=3, MAX_NUM_SEQS=1, MAX_BATCHED_TOKENS=2048
MAX_MODEL_LEN=512000, NUM_GPU_BLOCKS=2000, GPU_MEMORY_UTILIZATION=0.984
attention backend B12X_MLA_SPARSE, kv_cache_dtype kvarn_mla_k2_g64
cudagraph sizes 4,1; chunked prefill; no prefix caching

Keep /cache persistent — first boot JIT-compiles and later boots reuse it.

KV format ladder (KVarN)

Table with columns: Mode, dtype string, B/tok/layer, KLD
Mode	dtype string	B/tok/layer	KLD
K4V2 (default, champion)	kvarn_mla_k2_g64	290	0.0558
K4	kvarn_mla_k4_g64	418	0.0558
K5 (quality-max)	kvarn_mla_k5_g64	482	0.0554

K4V2 is the right-sized default: K4 buys zero measured KLD at +44% bytes. The exact-precision tail option was swept and measured neutral (speed and quality); the default is zero retained tail.

Provenance

Checkpoint: EXL3 ~3.40 bpw mixed-precision allocation assembled from per-projection cached-local-BF16-output recovery (expert_precision_map.json, projection-assembly.json in this folder). Base snapshot revision: b4734de4facf877f85769a911abafc5283eab3d9 (zai-org/GLM-5.2).
Runtime: local-inference-lab/vllm fork + b12x backend + KVarN + sparkinfer. Full source: github.com/JMPSequeira/glm52-kvarn-k4v2-runtime (the Docker image builds directly from that repo via its Containerfile; MANIFEST.md there maps every file to its upstream for PRs to local-inference-lab/vllm, b12x, and sparkinfer).
History: HISTORY.md in this folder (design decisions, KLDs, throughputs, rejected alternatives).
Measurement protocol notes: 5-minute post-boot settle; report prefill run index (first run of a pair reads ~3% fast from self-heating); same-session controls for prefill A/Bs.

Credits and references

Built on the open GLM-5.2 EXL3 serving community — the 3.5 bpw R7 and 3.42 bpw TR3 checkpoints and their deployment guides (brandonmusic, willfalco) defined the evidence standard this work follows.

KVarN — the variance-normalized KV-cache quantization this checkpoint's cache format builds on: paper arXiv:2606.03458, code huawei-csl/KVarN (native vLLM KV-cache backend).
b12x / sparkinfer — the sparse MLA backend and SM120 CuTe DSL kernel library serving this model (sparkinfer is b12x's current name).
local-inference-lab/vllm — the vLLM fork this runtime develops against.
Runtime source — JMPSequeira/glm52-kvarn-k4v2-runtime; image jmpsequeira/glm52-kvarn-k4v2:tr3 builds directly from it.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

jpsequeira

Model Tree

Base

zai-org/GLM-5.2

Quantized

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

What is different here

KVarN K4V2 KV cache — 290 B/token/layer latent-tile records with a self-describing dtype (kvarn_mla_k2_g64). Smaller than compact NVFP4 MLA KV (368 B) and roughly 2.3x smaller than FP8 (656 B), at higher measured fidelity than both on this model.
b12x (B12X_MLA_SPARSE) backend — sparse MLA attention with the paged top-k indexer, CKV gather prefill, and direct-packed DCP4 decode; deterministic top-k merge at zero measured cost.
local-inference-lab/vllm fork — the runtime base this stack is developed against; the serving image bakes the fork plus b12x, KVarN, and sparkinfer kernels into one bootable unit.

Checkpoint design: the MCG promotion ladder

The TR3 allocation was built in one evidence-driven pass, not uniform bits:

Full MCG3 base quantization — routed experts quantized to MCG3 using captures from the BF16 model on our own corpus.
Activation-aware promotion — candidates for promotion were selected by measured activation alignment on coding, reasoning, and agentic workloads (the domains this checkpoint is built for).
MCG4 as the promotion ceiling — aligned candidates were promoted MCG3 -> MCG4; nothing went higher because no measurement justified it.
Width milestones 3.00 -> 3.25 -> 3.38 -> 3.40 bpw — each step was validated, and 3.40 was the last balance point: pushing further would have forced cutting context capacity below the serving contract.
KVarN was a from-the-start design focus, not a retrofit — the KV format's size/quality potential shaped the whole budget, and the measured payoff (290 B/token at KLD 0.0558) is what makes 512K context coexist with the lowest published KLD.

Positioning vs the community checkpoints

Correction & retraction: KLD claim (2026-08-19)

Held-out measurement (wikitext-2 test window, 2,048 tokens, festr2 BF16 reference logits, fp8-KV reference per their methodology, same 3.40 bpw checkpoint varying only KV format):

Table with columns: KV cache, forward KLD (mean ± sd), top-1 agreement
KV cache	forward KLD (mean ± sd)	top-1 agreement
Standard FP8	0.0871 ± 0.0012	94.4%
KVarN K4V2 (this model's default)	0.0889 ± 0.0016	94.4%
KVarN K5	0.0900 ± 0.0012 (n=3)	94.3%
Dynamic NVFP4	0.1213 ± 0.0022	92.8%

What still stands:

KV economics: K4V2 costs +0.0016 over FP8; the community's Dynamic NVFP4 costs +0.0344 on the same weights — 20x the format charge. At 290 B/token vs 368, this is a smaller cache that is also more faithful.
Context: 819K proven (vs ~347K for nvfp4-KV deployments at this hardware class).
The item13 panel, plan SHA 072e37f..., and per-row token_ids_sha256 values are published for full reproduction of our calibration-domain numbers.

We intend to re-measure and, if a re-allocation campaign succeeds, re-enter the held-out comparison on these terms.

Quality evidence (calibration-domain — in-distribution; see the retraction above before quoting these numbers)

Direct BF16-reference KLD, full corpus item13 captures, 2,047 positions:

Table with columns: Configuration, KLD vs BF16
Configuration	KLD vs BF16
TR3 3.40 bpw + KVarN K4V2 KV	0.0558
TR3 3.40 bpw + KVarN K4 KV (418 B)	0.0558
TR3 3.40 bpw + KVarN K5 KV (482 B)	0.0554

Long-context: estonia-long 29/30 answers at 133K-token prompts.

Measured serving profile

TP4/DCP4/MTP3, one PCIe switch, 290 W locks, greedy:

Table with columns: Metric, Value
Metric	Value
Decode C1	86-91 tok/s (wall 32-34 ms, AL 2.9)
Decode c2 / c4 (256K pool)	104 / 168 tok/s
Prefill 8K-64K	2.4-2.5k tok/s
TTFT 32K	~13.5 s
Context	512K native (768K bootable)

Quick start

Runtime image (public): jmpsequeira/glm52-kvarn-k4v2:tr3 (all code baked — b12x, KVarN reader, sparkinfer, EXL3 loader).

IMPORTANT — re-pull the image if you downloaded it before 2026-08-19. Earlier pulls contain two prefix-caching bugs (KVarN KV + --enable-prefix-caching): garbage output on cache hits and persistent server-wide corruption after long requests. Fixed in tr3.2 (digest df889e27461a), which is also the current tr3. docker run does NOT auto-update a cached tag — run docker pull jmpsequeira/glm52-kvarn-k4v2:tr3 (delta download, seconds) or pin tr3.2. With the fix, prefix caching is safe and recommended; quality is unchanged (KLD 0.0555 vs 0.0558 of-record).

RESOLVED (2026-08-22) — Root cause found: not corruption. The 2-bit KV format (kvarn_mla_k2_g64) is below the generation-coherence floor. Every store/read kernel was verified mathematically sound (pack/rehydrate round-trip matches an independent reference exactly; the native SM120 decode reader agrees with record bytes to 0.3%; pool latents confirmed Hadamard-gaussianized; ownership/mapping verified valid at 55K probed decode steps). The 2-bit latent grid carries an inherent ~45% relative quantization error (measured; matches pure-RTN theory) — sufficient for prompt-side KL panels (short contexts are served from the FP8 exact pool and never touch packed rows — the 0.0558 KLD was an evaluation blind spot), but beyond what generation tolerates whenever quantized rows are attended: hot-window packing (PRECISION_TAIL_TOKENS=0) garbles immediately; old-row selection at larger tails garbles sporadically under history pressure. 4-bit (kvarn_mla_k4_g64, ~9% error) is clean in identical batteries when the hot window stays exact.

Layer-wise mixed precision — RETRACTED for serving (2026-08-23): the runtime supports demoting layers to the 2-bit record (KVARN_MLA_K2_LAYERS), and late layers (40-77) passed every standard gate (generation battery, tool calls, 30-turn soak, estonia-long 10/10 at 133K, cache-eviction churn). It is nonetheless too unstable for production: prefix-cache rehydration corrupts through the early layers' degenerate subnormal latent tiles, and extreme contexts (~700K) poison the serving boot outright. Until the format gains a story for those regimes, serve pure kvarn_mla_k4_g64 with KVARN_MLA_PRECISION_TAIL_TOKENS=3072.

k2-mix UPDATE (, 2026-08-26): the retraction's churn-corruption leg is disproven — it was the general records-ptr provenance wipe fixed in . Layers 40-77 demoted to on a k4 base pass the full churn soak post-fix (pressures, re-entries at full APC hit, exact needle retrieval). The k2-mix (~356 B/token, ~15% KV savings) is viable again for contexts ≤512K. The retraction now rests on a single remaining defect: ~700K contexts poison the serving boot (CKV lease issue, unfixed). The 2-bit record remains useful only for short-context, read-mostly workloads.

RESOLVED in tr3.4 (2026-08-26, digest f65645545938) — general cache-eviction corruption: the probabilistic, state-accumulating KV corruption under multi-100K eviction churn (all KV dtypes, worst under 2-bit mixes) was traced to a worker-side bookkeeping defect: a single shared records-pointer slot was alternated by per-layer rebinds and misread as record-storage reallocation, wiping flush provenance every rebind cycle and silently disabling record rehydrate on prefix-cache hits after eviction. Fixed by per-impl pointer keying; verified by a deterministic double-eviction reproducer and a 4-round churn soak (coherent re-entries at full 118K-token APC hit, exact needle retrieval). docker pull jmpsequeira/glm52-kvarn-k4v2:tr3.4 (also :latest). Earlier tags (tr3.3 and below) still carry the defect — the churn caveat applies to them only. This also re-attributes the 2-bit retraction's "rehydration corrupts through subnormal latent tiles" mechanism: that corruption was this bug; the 2-bit retraction stands on its remaining evidence (~700K boot poisoning, degraded gates).

tr3.5 (2026-08-26, digest 61ab86886a9c): fp8-RoPE record cell — opt-in. The KVarN packed record's RoPE rows move from BF16 to E4M3 + per-token fp16 amax (the #95 cell design): 418 → 356 B/token (−14.8%), ~15% more context per KV pool. Enable with KVARN_FP8_ROPE_RECORD=1. Validated end-to-end on the churn battery at 327680 ctx (pressures, 1s full-hit re-entries through fp8 records, exact needle retrieval, canary). Default OFF — tr3.5 without the flag is byte-identical serving behavior to tr3.4.

Serve with --kv-cache-dtype kvarn_mla_k4_g64 and KVARN_MLA_PRECISION_TAIL_TOKENS=3072 (the launcher defaults). The KVarN KLD table's 2-bit rows are valid only for short-context workloads. Separately filed: an MTP accepted-token-resolution race (fail-closed crash, not corruption), exposed under heavy per-step instrumentation — tracked in the runtime repo. root cause and regression suite once landed).

bash
docker run --rm --name glm52-kvarn \
  --gpus '"device=0,1,2,3"' --network host --ipc host \
  --ulimit memlock=-1:-1 --ulimit stack=67108864:67108864 \
  -v ~/.cache/huggingface:/hf-cache:ro \
  -v "$PWD/vllm-cache:/cache" \
  jmpsequeira/glm52-kvarn-k4v2:tr3 \
  --model jpsequeira/GLM-5.2-EXL3-TR3-3.40bpw-KVarN-K4V2 \
  ... (vllm serve flags per profile below)

Core profile env/flags (the full validated contract ships in the -src repo's boot-from-image.sh):

TP=4, DCP=4, MTP=3, MAX_NUM_SEQS=1, MAX_BATCHED_TOKENS=2048
MAX_MODEL_LEN=512000, NUM_GPU_BLOCKS=2000, GPU_MEMORY_UTILIZATION=0.984
attention backend B12X_MLA_SPARSE, kv_cache_dtype kvarn_mla_k2_g64
cudagraph sizes 4,1; chunked prefill; no prefix caching

Keep /cache persistent — first boot JIT-compiles and later boots reuse it.

KV format ladder (KVarN)

Table with columns: Mode, dtype string, B/tok/layer, KLD
Mode	dtype string	B/tok/layer	KLD
K4V2 (default, champion)	kvarn_mla_k2_g64	290	0.0558
K4	kvarn_mla_k4_g64	418	0.0558
K5 (quality-max)	kvarn_mla_k5_g64	482	0.0554

K4V2 is the right-sized default: K4 buys zero measured KLD at +44% bytes. The exact-precision tail option was swept and measured neutral (speed and quality); the default is zero retained tail.

Provenance

Checkpoint: EXL3 ~3.40 bpw mixed-precision allocation assembled from per-projection cached-local-BF16-output recovery (expert_precision_map.json, projection-assembly.json in this folder). Base snapshot revision: b4734de4facf877f85769a911abafc5283eab3d9 (zai-org/GLM-5.2).
Runtime: local-inference-lab/vllm fork + b12x backend + KVarN + sparkinfer. Full source: github.com/JMPSequeira/glm52-kvarn-k4v2-runtime (the Docker image builds directly from that repo via its Containerfile; MANIFEST.md there maps every file to its upstream for PRs to local-inference-lab/vllm, b12x, and sparkinfer).
History: HISTORY.md in this folder (design decisions, KLDs, throughputs, rejected alternatives).
Measurement protocol notes: 5-minute post-boot settle; report prefill run index (first run of a pair reads ~3% fast from self-heating); same-session controls for prefill A/Bs.

Credits and references

Built on the open GLM-5.2 EXL3 serving community — the 3.5 bpw R7 and 3.42 bpw TR3 checkpoints and their deployment guides (brandonmusic, willfalco) defined the evidence standard this work follows.

KVarN — the variance-normalized KV-cache quantization this checkpoint's cache format builds on: paper arXiv:2606.03458, code huawei-csl/KVarN (native vLLM KV-cache backend).
b12x / sparkinfer — the sparse MLA backend and SM120 CuTe DSL kernel library serving this model (sparkinfer is b12x's current name).
local-inference-lab/vllm — the vLLM fork this runtime develops against.
Runtime source — JMPSequeira/glm52-kvarn-k4v2-runtime; image jmpsequeira/glm52-kvarn-k4v2:tr3 builds directly from it.

GLM-5.2-EXL3-TR3-3.40bpw-KVarN-K4V2

README

What is different here

Checkpoint design: the MCG promotion ladder

Positioning vs the community checkpoints

Correction & retraction: KLD claim (2026-08-19)

Quality evidence (calibration-domain — in-distribution; see the retraction above before quoting these numbers)

Measured serving profile

Quick start

KV format ladder (KVarN)

Provenance

Credits and references

Explore FriendliAI today

README

What is different here

Checkpoint design: the MCG promotion ladder

Positioning vs the community checkpoints

Correction & retraction: KLD claim (2026-08-19)

Quality evidence (calibration-domain — in-distribution; see the retraction above before quoting these numbers)

Measured serving profile

Quick start

KV format ladder (KVarN)

Provenance

Credits and references