malaiwah

GLM-5.3-Flash-TR3-8bpw

Quality — SEALED, full panel, two bitwise-identical cold runs

⚠ Scope disclosure — this number is a panel25 number

Added 2026-08-29. Nothing here is a correction: 0.012384 is and remains the correct mean over the full 25-window panel. What changed is that the panel is now known to contain calibration-adjacent windows, so the scope has to travel with the number.

brandonmusic ran a 13-gram overlap scan of his sealed panel against its own calibration-role windows and found that the whole axis4_reasoning domain shares 37–39 % of its 13-grams with calibration material — despite the panel being clean at the document-hash level. Document-hash dedup is not enough. He excluded that domain and scored his primary numbers on the 17 windows that survive. The finding, the scan and the 0.05 threshold are his.

Every malaiwah number on this panel used all 25 windows, so every one of them carries the same contamination. Recomputed on his clean scope, from our own published per-window arrays (no GPU, no re-measurement — this is arithmetic on data already published):

Table with columns: panel25 (published), clean17 (his scope), move
panel25 (published) clean17 (his scope) move
K8 0.012384 0.010829 −12.55 %
K6 sealed 0.013723 0.011677 −14.91 %
K6 streaming 0.013715 0.011676 −14.87 %
official FP8 0.020615 0.018665 −9.46 %
BF16 floor (cross-stack) 0.012712 0.010648 −16.24 %
brandonmusic 4bpw 0.024555

The comparisons hold, and one of them weakens. K8 beats the official FP8 on 17 of 17 clean windows, and the margin widens: 1.66× on panel25 becomes 1.72× on clean17. The K8-over-K6 result survives but weakens — the paired BCa interval still excludes zero, but its lower bound falls from +0.000695 to +0.000153 and the sign test goes from p = 0.0041 to p = 0.049. We will not restate "K8 is better than K6" without naming the scope.

Do not difference a panel25 number against a clean17 one. They are answers to different questions. Our registry enforces this structurally: clean17 is its own derived panel with its own comparability key.

The quantization-attributable table below cannot be recomputed on the clean scope — its floor is the streaming BF16 floor, whose receipt is scalar-only (run means and a tokenwise digest, no per-window array), and substituting the cross-stack floor would be the cross-lane subtraction our registry refuses. It stands as a panel25 number.

Full recompute, with per-domain tables, paired intervals and provenance: reports/clean-scope-recompute.json. Working: PROTOCOL-ALIGNMENT.md §4.

One protocol note, not a correction. His protocol masks the 24 padded lm_head columns before the log-softmax; ours never has. Measured on his real teacher window, the padded columns hold ~1.6e-8 of the probability mass, and because this quant shares the teacher's native BF16 head the effect collapses to KLD × mass — 1.0e-10 nats, moving the value above at its 9th significant figure. For scale, our own sealed-vs-streaming bridge is 8.5e-6 and the window-clustered SE on this panel is 3.19e-3. No correction and no bias disclosure is warranted; we are adopting masking anyway. Script and receipts: bin/padded_column_study.py.

Table with columns: panel25 (published), clean17 (his scope), move
	panel25 (published)	clean17 (his scope)	move
K8	0.012384	0.010829	−12.55 %
K6 sealed	0.013723	0.011677	−14.91 %
K6 streaming	0.013715	0.011676	−14.87 %
official FP8	0.020615	0.018665	−9.46 %
BF16 floor (cross-stack)	0.012712	0.010648	−16.24 %
brandonmusic 4bpw	0.024555

Mean KLD(teacher ‖ K8) = 0.012384191023436866 over the full sealed panel (25 windows / 51,175 positions), two cold runs producing identical means to the last digit (bitwise_deterministic: true). Quality gate passed. Receipt: receipts/stream-k8-kld.json.

Table with columns: Model, Mean KLD (nats), Size, Lane
Model	Mean KLD (nats)	Size	Lane
This K8	0.012384	331 GB	streaming, 2 runs
K6	0.013715	254 GB	streaming, 2 runs
K6 (sealed 8×H200, 5 runs)	0.013723	254 GB	sealed EP8
Official FP8	0.020615

At the same footprint as the official FP8 release (331 vs 328 GB), K8 is 1.66× closer to the BF16 teacher — and 1.11× closer than K6 at 30 % more bytes. Weight-space corroboration: with the intermediate-channel permutation undone, K8's shipped store is 13.2× tighter in NMSE than K6's (3.505e-5 vs 4.624e-4, better in 30 of 30 sampled matrices).

Lane disclosure. Measured on the single-GPU streaming lane (~$6/model), not the 8×H200 sealed lane. The lanes were bridged on this exact panel: K6 reads 0.013714889 streaming vs 0.013723385 sealed — −8.5e-6 (0.06 %), with the worst single window differing by 2.9e-4. The streaming receipt sets publishable_as_reproduction: false because a different expert-combine order is an independent measurement that agrees closely, not a bitwise reproduction.

Methodology note worth stealing. A single-window comparison of these two rates is statistically meaningless: per-window KLD scatter has sd 1.73e-3 against a K6-vs-K8 effect of 1.22e-3. On one unlucky window (window-0000) K8 appeared worse than K6; over the full panel it wins decisively. Never quote a single-window KLD as a rate comparison — full write-up.

Quantization-attributable error (the floor removed)

Scoring the unquantized BF16 weights against this teacher on this panel already costs 0.011506 nats — the price of the comparison itself (teacher captured on a different runtime; bf16 addition is not associative across differing expert-combine orders). Two cold runs, identical means. Removing it:

Table with columns: panel KLD, attributable to quantization
	panel KLD	attributable to quantization
BF16 (floor)	0.011506	—
K8 (331 GB)	0.012384	0.000878
K6 (254 GB)	0.013715	0.002209

K8's quantization error is 2.52x smaller than K6's, against a raw ratio of only 1.11x — K8 removes ~60% of the divergence K6 leaves behind. Raw KLD understates differences between good quants because the floor is common to both. Method, receipts and the ways this subtraction can be misused: BF16-FLOOR.md.

What this is (and is not)

Codec: EXL3-format TR3/MCG trellis (turboderp's exllamav3 kernels @ c5d9c657), through brandonmusic's GLM-5.3 pipeline with a disclosed patch series. His published core admits K3/K4/K5, so K8 is a declared rate extension — our encoder was verified byte-identical to his sealed core across 120 encodes / 624 MiB / 0 differing bytes (evidence, issue #1).
Serving runtime: use malaiwah/glm52-exl3-vast with MODEL_PROFILE=glm53-k8. The image pins the qualified Glm5Next vLLM, B12X sparse-attention stack, native EXL3 K8 extension, CUDA, and fail-closed runtime overlays as one contract.
Not stock exllamav3/TabbyAPI or stock upstream vLLM: those stacks do not carry this complete + TR3/MCG K8 serving path.

Provenance & disclosed deviations

Pins: BF16 source zai-org/GLM-5.3-Flash-BF16 (weights == a6c167b6); calibration = brandonmusic's published EP4 captures (sealed inventory f56e9d62… adopted verbatim). Deviations, all receipted: encoded on 4×H200 SM90 (his campaign attests 4×B200 SM100; fat 9.0;10.0 extension build), K4-KL gate satisfied via a disclosed bridge document carrying his real published K4 receipt hashes, measurement on the streaming lane at EP8 emulation with fp32 combine order. Materialization receipt: bits 8, complete, main_and_mtp_complete, nonrouted_native_exact, 331,449,761,784 logical bytes, 37,152 routed choices, 1,618 native tensors.

Lineage on the Hub

Z.ai published two sibling roots for this model and neither declares the other: zai-org/GLM-5.3-Flash (the FP8 release, where most traffic lands) and zai-org/GLM-5.3-Flash-BF16 (the BF16 weights). This quant declares BF16 as its base_model because that is what it was actually quantized from — the FP8 release is a sibling quantization of the same model, not our source, and it is the baseline we measure against rather than build on. Quants that list FP8 as their base were genuinely made from the FP8 weights; the trees differ for real reasons.

Related work on the same model, all measured on one panel in the quant-fidelity registry: brandonmusic 4bpw, 0xSero Dione Q4, orcarouter MLX. Collection: GLM-5.3-Flash — measured quants & fidelity.

Credits

Base model by Z.ai. Quantization pipeline, calibration captures, and teacher panel by brandonmusic — co-credited, see the collaboration thread. Trellis codec and kernels by turboderp. Every tool, patch, receipt and the full campaign log: malaiwah/glm53-flash-fidelity-suite. Comparable measurements across quants: quant-fidelity-registry.

Serving — live-qualified turnkey profile

Qualification result

The shipped profile is glm53-k8. It was booted on 4× RTX PRO 6000 Blackwell 96 GiB and passed arithmetic, factual, instruction-following, strict structured-output, and tokenizer-exact 32K retrieval startup gates. The final published image was then pulled by digest and smoke-tested without source or runtime-overlay mounts.

Appliance source commit: a0d05f76994cf44f3667c0d2910d3b0e4d305d23
Checkpoint revision: b5ef443adce36ba5a10f2d5aa682fc9f2f0d0fae
Qualified parent: verdictai/glm53-flash-exl3-k4@sha256:0f1cdcc8891f1cc3a444121eb61d366289a1cbba285f0892dcbb24bc94961692
Published appliance: ghcr.io/malaiwah/glm52-exl3-vast@sha256:5a0d4b370e9f6a2ef85fa8b8c213492122554b34ba18d630a3a78130758914cf
Shape: TP4 / DCP4 A2A, B12X sparse MLA, Triton MoE, native EXL3 K8, calibrated NVFP4-DS MLA KV, MTP off, eager mode, batch 512, C8, GMU 0.93
Request limit: 458,752 tokens; text-only qualification scope

Why K8 uses the native eager path

The eight overlapping 16-bit MCG windows for K8 span 72 bits, while B12X's fused EXL3 decoder represents that state in two 64-bit words. The runtime therefore fails closed instead of merely widening the fused decoder's bitrate guard. This profile uses ExLlamaV3's compiled native K8 extension and passes the actual uniform layer bitrate into the K8 dispatch. VLLM_EXL3_PREFILL_CAPACITY is bounded to the scheduler's 512-token batch and eager mode avoids an unqualified graph path. B12X sparse MLA and the rest of the qualified GLM-5.3 stack remain enabled.

The engine exposed 6,610,733 logical KV tokens, or 14.41 maximum-length requests. Per-GPU profiling reported 78.94–78.97 GiB weights plus non-torch, 2.25 GiB peak activations, zero graph memory, and 7.10–7.14 GiB KV.

Correctness and context

Two independent 448K trials each built a tokenizer-exact 449,461-token document and retrieved all three facts, in 170.775 and 175.086 seconds. Short and strict structured-output checks still passed after the long-context stress. The common 458,752-token K6/K8 cap is a correctness boundary, not an extrapolation from KV capacity.

Measured throughput

Unique-prefix prefill, one request, no prefix reuse:

Table with columns: prompt, tok/s
prompt	tok/s
8K	2,684
32K	2,825
64K	2,938
128K	2,986

Aggregate target-only decode (MTP_TOKENS=0), eight requests per level:

Table with columns: input context, C1 tok/s, C4 tok/s, C8 tok/s
input context	C1 tok/s	C4 tok/s	C8 tok/s
256	10.29	38.79	75.66
32K	8.42	23.43	28.46
128K	5.42	12.05	13.25

All 72 requests completed without failure, preemption, or prefix reuse. These are measurements from one four-GPU PCIe host, not guarantees for other topology, clock, thermal, driver, storage, or request mixes.

K8 versus the K6 production default

K8 lowers panel KLD from K6's 0.013723 to 0.012384. After subtracting the common BF16/runtime floor, its attributable quantization error is 2.52× smaller. The cost is about 77 GB / 30% more checkpoint bytes, about 15.2 GiB more non-KV memory per GPU, and about 14.4 GiB less KV per GPU. K6 is 5.3–7.3× faster at short context and 8.3–24.4× faster when the measured 32K/128K prefill cost is included. K8 is the fidelity-first option; K6 remains the production default.

Docker Compose

Prerequisites: Linux x86-64, four visible RTX PRO 6000 Blackwell GPUs, NVIDIA driver ≥ 590.48.01 / CUDA 13.2 compatibility, the NVIDIA Container Toolkit, and roughly 400 GiB free persistent storage for the checkpoint plus caches. PCIe P2P on this card family requires NVIDIA's open kernel modules; see the RTX 6000 Pro multi-GPU notes.

yaml
name: glm53-k8
services:
  api:
    image: ghcr.io/malaiwah/glm52-exl3-vast@sha256:5a0d4b370e9f6a2ef85fa8b8c213492122554b34ba18d630a3a78130758914cf
    pull_policy: always
    restart: unless-stopped
    network_mode: host
    ipc: host
    shm_size: 32gb
    stop_grace_period: 2m
    ulimits:
      memlock:
        soft: -1
        hard: -1
    environment:
      MODEL_PROFILE: glm53-k8
      AUTH: key
      VLLM_API_KEY: ${VLLM_API_KEY:?set VLLM_API_KEY to a long random secret}
      HF_TOKEN: ${HF_TOKEN:-}
      SSH_ENABLED: "0"
      SOUL_ENABLED: "0"
      VERIFY_HEALTH_TIMEOUT_S: "3600"
    volumes:
      - /srv/glm53-turnkey:/workspace
      - /srv/glm53-cache:/cache
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 4
              capabilities: [gpu]

bash
sudo mkdir -p /srv/glm53-turnkey /srv/glm53-cache
export VLLM_API_KEY="$(openssl rand -hex 32)"
docker compose up -d
docker compose logs -f

First boot downloads about 309 GiB and can take substantial time. The container is ready only after the log reports

markdown

>>> Verified: serving; long-context retrieval verified

. API: http://HOST:8000/v1; dashboard: http://HOST:1111. The served model name is GLM-5.3-Flash-K8.

bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Authorization: Bearer $VLLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"GLM-5.3-Flash-K8","messages":[{"role":"user","content":"Reply with exactly READY"}],"max_tokens":256}'

Do not replace only the checkpoint path in another vLLM command. The profile, parent digest, runtime overlays, quantization, attention backend, DCP topology, KV calibration, scheduler, and eager execution mode are one qualified contract.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

malaiwah

Model Tree

Base

zai-org/GLM-5.3-Flash-BF16

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

Quality — SEALED, full panel, two bitwise-identical cold runs

⚠ Scope disclosure — this number is a panel25 number

Added 2026-08-29. Nothing here is a correction: 0.012384 is and remains the correct mean over the full 25-window panel. What changed is that the panel is now known to contain calibration-adjacent windows, so the scope has to travel with the number.

brandonmusic ran a 13-gram overlap scan of his sealed panel against its own calibration-role windows and found that the whole axis4_reasoning domain shares 37–39 % of its 13-grams with calibration material — despite the panel being clean at the document-hash level. Document-hash dedup is not enough. He excluded that domain and scored his primary numbers on the 17 windows that survive. The finding, the scan and the 0.05 threshold are his.

Every malaiwah number on this panel used all 25 windows, so every one of them carries the same contamination. Recomputed on his clean scope, from our own published per-window arrays (no GPU, no re-measurement — this is arithmetic on data already published):

Table with columns: panel25 (published), clean17 (his scope), move
panel25 (published) clean17 (his scope) move
K8 0.012384 0.010829 −12.55 %
K6 sealed 0.013723 0.011677 −14.91 %
K6 streaming 0.013715 0.011676 −14.87 %
official FP8 0.020615 0.018665 −9.46 %
BF16 floor (cross-stack) 0.012712 0.010648 −16.24 %
brandonmusic 4bpw 0.024555

The comparisons hold, and one of them weakens. K8 beats the official FP8 on 17 of 17 clean windows, and the margin widens: 1.66× on panel25 becomes 1.72× on clean17. The K8-over-K6 result survives but weakens — the paired BCa interval still excludes zero, but its lower bound falls from +0.000695 to +0.000153 and the sign test goes from p = 0.0041 to p = 0.049. We will not restate "K8 is better than K6" without naming the scope.

Do not difference a panel25 number against a clean17 one. They are answers to different questions. Our registry enforces this structurally: clean17 is its own derived panel with its own comparability key.

The quantization-attributable table below cannot be recomputed on the clean scope — its floor is the streaming BF16 floor, whose receipt is scalar-only (run means and a tokenwise digest, no per-window array), and substituting the cross-stack floor would be the cross-lane subtraction our registry refuses. It stands as a panel25 number.

Full recompute, with per-domain tables, paired intervals and provenance: reports/clean-scope-recompute.json. Working: PROTOCOL-ALIGNMENT.md §4.

One protocol note, not a correction. His protocol masks the 24 padded lm_head columns before the log-softmax; ours never has. Measured on his real teacher window, the padded columns hold ~1.6e-8 of the probability mass, and because this quant shares the teacher's native BF16 head the effect collapses to KLD × mass — 1.0e-10 nats, moving the value above at its 9th significant figure. For scale, our own sealed-vs-streaming bridge is 8.5e-6 and the window-clustered SE on this panel is 3.19e-3. No correction and no bias disclosure is warranted; we are adopting masking anyway. Script and receipts: bin/padded_column_study.py.

Table with columns: panel25 (published), clean17 (his scope), move
	panel25 (published)	clean17 (his scope)	move
K8	0.012384	0.010829	−12.55 %
K6 sealed	0.013723	0.011677	−14.91 %
K6 streaming	0.013715	0.011676	−14.87 %
official FP8	0.020615	0.018665	−9.46 %
BF16 floor (cross-stack)	0.012712	0.010648	−16.24 %
brandonmusic 4bpw	0.024555

Table with columns: Model, Mean KLD (nats), Size, Lane
Model	Mean KLD (nats)	Size	Lane
This K8	0.012384	331 GB	streaming, 2 runs
K6	0.013715	254 GB	streaming, 2 runs
K6 (sealed 8×H200, 5 runs)	0.013723	254 GB	sealed EP8
Official FP8	0.020615

Quantization-attributable error (the floor removed)

Table with columns: panel KLD, attributable to quantization
	panel KLD	attributable to quantization
BF16 (floor)	0.011506	—
K8 (331 GB)	0.012384	0.000878
K6 (254 GB)	0.013715	0.002209

What this is (and is not)

Codec: EXL3-format TR3/MCG trellis (turboderp's exllamav3 kernels @ c5d9c657), through brandonmusic's GLM-5.3 pipeline with a disclosed patch series. His published core admits K3/K4/K5, so K8 is a declared rate extension — our encoder was verified byte-identical to his sealed core across 120 encodes / 624 MiB / 0 differing bytes (evidence, issue #1).
Serving runtime: use malaiwah/glm52-exl3-vast with MODEL_PROFILE=glm53-k8. The image pins the qualified Glm5Next vLLM, B12X sparse-attention stack, native EXL3 K8 extension, CUDA, and fail-closed runtime overlays as one contract.
Not stock exllamav3/TabbyAPI or stock upstream vLLM: those stacks do not carry this complete + TR3/MCG K8 serving path.

Provenance & disclosed deviations

Lineage on the Hub

Related work on the same model, all measured on one panel in the quant-fidelity registry: brandonmusic 4bpw, 0xSero Dione Q4, orcarouter MLX. Collection: GLM-5.3-Flash — measured quants & fidelity.

Credits

Serving — live-qualified turnkey profile

Qualification result

Appliance source commit: a0d05f76994cf44f3667c0d2910d3b0e4d305d23
Checkpoint revision: b5ef443adce36ba5a10f2d5aa682fc9f2f0d0fae
Qualified parent: verdictai/glm53-flash-exl3-k4@sha256:0f1cdcc8891f1cc3a444121eb61d366289a1cbba285f0892dcbb24bc94961692
Published appliance: ghcr.io/malaiwah/glm52-exl3-vast@sha256:5a0d4b370e9f6a2ef85fa8b8c213492122554b34ba18d630a3a78130758914cf
Shape: TP4 / DCP4 A2A, B12X sparse MLA, Triton MoE, native EXL3 K8, calibrated NVFP4-DS MLA KV, MTP off, eager mode, batch 512, C8, GMU 0.93
Request limit: 458,752 tokens; text-only qualification scope

Why K8 uses the native eager path

Correctness and context

Measured throughput

Unique-prefix prefill, one request, no prefix reuse:

Table with columns: prompt, tok/s
prompt	tok/s
8K	2,684
32K	2,825
64K	2,938
128K	2,986

Aggregate target-only decode (MTP_TOKENS=0), eight requests per level:

Table with columns: input context, C1 tok/s, C4 tok/s, C8 tok/s
input context	C1 tok/s	C4 tok/s	C8 tok/s
256	10.29	38.79	75.66
32K	8.42	23.43	28.46
128K	5.42	12.05	13.25

K8 versus the K6 production default

Docker Compose

yaml
name: glm53-k8
services:
  api:
    image: ghcr.io/malaiwah/glm52-exl3-vast@sha256:5a0d4b370e9f6a2ef85fa8b8c213492122554b34ba18d630a3a78130758914cf
    pull_policy: always
    restart: unless-stopped
    network_mode: host
    ipc: host
    shm_size: 32gb
    stop_grace_period: 2m
    ulimits:
      memlock:
        soft: -1
        hard: -1
    environment:
      MODEL_PROFILE: glm53-k8
      AUTH: key
      VLLM_API_KEY: ${VLLM_API_KEY:?set VLLM_API_KEY to a long random secret}
      HF_TOKEN: ${HF_TOKEN:-}
      SSH_ENABLED: "0"
      SOUL_ENABLED: "0"
      VERIFY_HEALTH_TIMEOUT_S: "3600"
    volumes:
      - /srv/glm53-turnkey:/workspace
      - /srv/glm53-cache:/cache
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 4
              capabilities: [gpu]

bash
sudo mkdir -p /srv/glm53-turnkey /srv/glm53-cache
export VLLM_API_KEY="$(openssl rand -hex 32)"
docker compose up -d
docker compose logs -f

First boot downloads about 309 GiB and can take substantial time. The container is ready only after the log reports

markdown

>>> Verified: serving; long-context retrieval verified

. API: http://HOST:8000/v1; dashboard: http://HOST:1111. The served model name is GLM-5.3-Flash-K8.

bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Authorization: Bearer $VLLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"GLM-5.3-Flash-K8","messages":[{"role":"user","content":"Reply with exactly READY"}],"max_tokens":256}'

GLM-5.3-Flash-TR3-8bpw

README

Quality — SEALED, full panel, two bitwise-identical cold runs

⚠ Scope disclosure — this number is a panel25 number

Quantization-attributable error (the floor removed)

What this is (and is not)

Provenance & disclosed deviations

Lineage on the Hub

Credits

Serving — live-qualified turnkey profile

Qualification result

Why K8 uses the native eager path

Correctness and context

Measured throughput

K8 versus the K6 production default

Docker Compose

Explore FriendliAI today

README

Quality — SEALED, full panel, two bitwise-identical cold runs

⚠ Scope disclosure — this number is a panel25 number

Quantization-attributable error (the floor removed)

What this is (and is not)

Provenance & disclosed deviations

Lineage on the Hub

Credits

Serving — live-qualified turnkey profile

Qualification result

Why K8 uses the native eager path

Correctness and context

Measured throughput

K8 versus the K6 production default

Docker Compose