malaiwah

GLM-5.3-Flash-TR3-6bpw

QFS size–fidelity evidence

The panel25 streaming-lane measurements: 25 windows, 51,175 scored positions, full-vocabulary KL(reference || candidate) in nats, with this artifact highlighted and its peers retained. K6 is plotted at 0.013714889 nats and K8 at 0.012384191 nats; these are the two-run streaming receipts, not K6's five-run sealed-ep8 result. Neither clean17 values nor cross-stack FP8 measurements are mixed into this plot.

This is inspection-only, not a certified ranking: the registry retains its scope, pipeline and missing-provenance limitations and draws no ranking line. The x-axis is recorded serialized bytes in GiB, not VRAM; K6/K8 use tensor-payload bytes while peers can use whole-repository bytes. The unquantized control has no recorded size, so it remains in the downloadable data with its exclusion reason rather than receiving an invented x-coordinate. No control subtraction is performed.

These are descriptive means on the historical panel: its 25 windows come from four source documents and include calibration-adjacent material. They do not establish population-level, native-serving or task-quality rankings; see the scope disclosures below.

Interactive highlighted plot · PNG · SVG · CSV and exclusions · Full provenance JSON · Live cached image

Registry snapshot: 598c441a2281963f1469ea4ec02d166081b3ac5a. The static plot and data are stored with this card; the live image is explicitly mutable. No model weights or measurement values were changed by this plot addition.

Fidelity — sealed-ep8, five-cold-run qualification

⚠ Scope disclosure — this number is a panel25 number

Added 2026-08-29. Nothing here is a correction: 0.013723 is and remains the correct mean over the full 25-window panel. What changed is that the panel is now known to contain calibration-adjacent windows, so the scope has to travel with the number.

brandonmusic ran a 13-gram overlap scan of his sealed panel against its own calibration-role windows and found that the whole axis4_reasoning domain shares 37–39 % of its 13-grams with calibration material — despite the panel being clean at the document-hash level. Document-hash dedup is not enough. He excluded that domain and scored his primary numbers on the 17 windows that survive. The finding, the scan and the 0.05 threshold are his.

Every malaiwah number on this panel used all 25 windows, so every one of them carries the same contamination. Recomputed on his clean scope, from our own published per-window arrays (no GPU, no re-measurement — this is arithmetic on data already published):

Table with columns: panel25 (published), clean17 (his scope), move
panel25 (published) clean17 (his scope) move

Mean KLD(teacher ‖ K6) = 0.013723 nats over the full sealed panel (25 windows, 51,175 positions per run) — five cold runs with matching recorded tokenwise-KL digests and run means (population stddev exactly 0.0). This is conditional repeatability, not universal determinism or independent textual replication. The receipt's quality gate (< 0.06) passed. Receipts: receipts/k6-five-run-kld.json, receipts/k6-packed-kld.json (evidence-artifact hashes included).

The following raw values are a mixed-design inventory, not a ranking. Equal panel identity is necessary; the actual pair predicate must also pass.

Table with columns: Model, Mean KLD (nats), Size, Scope
Model	Mean KLD (nats)	Size	Scope
This K6 (sealed)	0.013723	254 GB	full panel × 5 bitwise-identical runs
This K6, streaming lane	0.013715	254 GB	full panel × 2 bitwise-identical runs; −8.5e-6 vs sealed (receipt)
K8 sibling	0.012384	331 GB

Historical single-window diagnostics, not comparable to the panel means:

Table with columns: Model, Mean KLD (nats), Size, Scope
Model	Mean KLD (nats)	Size	Scope
Official FP8 (his stack, v44)	0.024629	328 GB	1 window × 5 runs
NVFP4 (his stack, v44)	0.060535	~180 GB	1 window × 5 runs

Correction, 2026-09-07. The former "1.5× lower divergence" FP8, "1.8× vs 4bpw" and "4.4× vs NVFP4" headlines are withdrawn: they mix runtime/lane and, for NVFP4, one window with a full panel. Panel-wide top-1 remains 96.56 % (25-window panel, streaming lane). The separate serving qualification below is evidence for its pinned deployment, not equivalence to this reference-forward fidelity measurement or a current remote-status check.

The checkpoint measurements reconstruct weights through the reference forward; they do not measure native-serving activation/cache/kernel numerics, or prove native decode/forward equivalence. Weights-only KL is not a lower bound on served KL: omitted perturbations may amplify or cancel divergence.

Excess over control (descriptive subtraction)

(Called "quantization-attributable error" before 2026-08-31; renamed per peer-review P1-05 — the difference estimates excess divergence over the same-lane unquantized control and is not a causal attribution.)

Scoring the unquantized BF16 weights against this teacher on this panel already costs 0.011506 nats — the price of the comparison itself (teacher captured on a different runtime; bf16 addition is not associative across differing expert-combine orders). Two cold runs, identical means. Subtracting it:

Table with columns: panel KLD, excess over control
	panel KLD	excess over control
BF16 (floor)	0.011506	—
K8 (331 GB)	0.012384	0.000878
K6 (254 GB)	0.013715	0.002209

K8's residual is smaller than K6's — 0.000878 vs 0.002209 nats — where the raw means sit only 1.11x apart, because the floor is common to both rows and dominates both. The previously published ratio of the two residuals ("2.52x") is withdrawn: a ratio of small residuals magnifies control error and carried no uncertainty. Read the residuals beside the raw values, with the floor named. Method, receipts and the ways this subtraction can be misused: BF16-FLOOR.md.

What this is (and is not)

Codec: EXL3-format TR3/MCG trellis (turboderp's exllamav3 kernels @ c5d9c657 did the encoding math), through brandonmusic's GLM-5.3 quantization pipeline with a small disclosed patch series.
Serving runtime: use malaiwah/glm52-exl3-vast with MODEL_PROFILE=glm53-k6. The image pins the qualified Glm5Next vLLM, B12X, EXL3, CUDA, and 21-file fail-closed runtime overlay as one contract.
Not stock exllamav3/TabbyAPI or stock upstream vLLM: those stacks do not carry this complete glm5_next + TR3/MCG K6 serving path.
Topology-neutral checkpoint: canonical unsharded tensors; TP layout is a load-time decision. The qualified deployment is exactly TP4/DCP4 on four 96 GiB RTX PRO 6000 Blackwell GPUs.
Measured memory: the packaged image loads 58.96 GiB of model tensors per rank. At GMU 0.93, final profiling reported 63.74–63.78 GiB weights + non-torch, 3.02 GiB peak activations, 0.45–0.46 GiB CUDA graphs, and 21.52–21.56 GiB KV per GPU.

Provenance & disclosed deviations

Full receipts ship in this repo and in the fidelity suite. Key pins: BF16 source zai-org/GLM-5.3-Flash-BF16 (weights == a6c167b6), calibration = brandonmusic's published EP4 captures (sealed inventory f56e9d62… adopted verbatim), same transform-seed discipline as the K8 sibling (parts-bin assembly compatible). Disclosed deviations from his sealed K4 campaign: encoded on 4×H200 SM90 (his: 4×B200 SM100; fat 9.0;10.0 extension build), verified-equivalent R10 codec (we encoded with a reconstruction while his numeric core was unpublished; he has since published the sealed closure, and a head-to-head on identical real inputs came back 120/120 encodes byte-identical — 624 MiB of packed trellis, 0 differing bytes, decoded-weight delta exactly 0.0. His published core admits only K3/K4/K5, so K6/K8 are a declared rate extension, not a substitution; driving his sealed primitives past that admission constant reproduces our bytes on those sampled cases. Full-campaign encoded identity and zero full-campaign fidelity impact were not established. Evidence: closure-comparison.json, issue #1), K4-KL gate satisfied via a disclosed bridge document carrying his real published K4 receipt hashes, qualification at EP8 (his reader default EP4). The five-run qualification receipts land here when sealed.

Family

Table with columns: K4, K6 (this), K8
	K4	K6 (this)	K8
Repo	brandonmusic's 4bpw	here	malaiwah/GLM-5.3-Flash-TR3-8bpw
Size	176 GB	254 GB	331 GB
Mean KLD (same panel)	0.024555	0.013723	0.012384

The family shares source/calibration lineage and panel, not one measurement pipeline. A payload parts-bin dataset (K6 + K8 per-choice payloads, same seed) is recorded as published: GLM-5.3-Flash-TR3-partsbin-v1. Mixed-rate assembly is decodable but does not reproduce a fresh encode with down-projection Hessians conditioned on that mixed gate/up choice; its fidelity requires its own measurement.

Lineage on the Hub

Z.ai published two sibling roots for this model and neither declares the other: zai-org/GLM-5.3-Flash (the FP8 release, where most traffic lands) and zai-org/GLM-5.3-Flash-BF16 (the BF16 weights). This quant declares BF16 as its base_model because that is what it was actually quantized from — the FP8 release is a sibling quantization of the same model, not our source. The fidelity reference here is the pinned BF16 teacher; FP8 is a separately measured cross-stack candidate, not the reference. A declared base-model link alone does not prove another publisher's weight provenance.

Related work on the same model, all measured on one panel in the quant-fidelity registry: brandonmusic 4bpw, 0xSero Dione Q4, orcarouter MLX. Collection: GLM-5.3-Flash — measured quants & fidelity.

Credits

Base model by Z.ai. Quantization pipeline, calibration captures, and teacher panel by brandonmusic (co-credited — see the collaboration thread). Trellis codec and kernels by turboderp. Campaign log, tools, and every patch: malaiwah/quant-fidelity-suite.

Serving — live-qualified turnkey profile

Qualification result

The shipped profile is glm53-k6. It was booted from the packaged appliance on 4× RTX PRO 6000 Blackwell 96 GiB and passed the appliance's arithmetic, factual, instruction-following, strict structured-output, and tokenizer-exact 32K retrieval gates. Runtime fingerprint: vllm-0.1.dev20111+g7f1e92bec.d20260827-tp4-95ae22a9.

Appliance source commit: a0d05f76994cf44f3667c0d2910d3b0e4d305d23
Qualified parent: verdictai/glm53-flash-exl3-k4@sha256:0f1cdcc8891f1cc3a444121eb61d366289a1cbba285f0892dcbb24bc94961692
Published appliance: ghcr.io/malaiwah/glm52-exl3-vast@sha256:5a0d4b370e9f6a2ef85fa8b8c213492122554b34ba18d630a3a78130758914cf
Shape: TP4 / DCP4 A2A, B12X sparse MLA, Triton MoE, EXL3 K6, calibrated NVFP4-DS MLA KV, MTP off, batch 3,072, C8, GMU 0.93
Request limit: 458,752 tokens; text-only qualification scope

The cap is a correctness boundary, not a memory-capacity guess. Two independent 448K trials produced tokenizer-exact 449,461- and 449,462-token documents and retrieved 3/3 facts at 15%, 55%, and 90% depth. A 480K trial exhausted both 2,048- and 4,096-token answer budgets. A concurrent 505K stress trial caused persistent degenerate follow-on output until restart. Therefore this release does not claim a 500K usable request. The 458,752-token envelope leaves about 9K tokens beyond the longest passing document for template, query, and output.

Table with columns: document tokens, independent seed, retrieval, elapsed
document tokens	independent seed	retrieval	elapsed
384,612	20260831	3/3	81.058 s
449,462	20260901	3/3	95.839 s
449,461	20260903	3/3	94.578 s

The final appliance boot auto-profiled 20,043,933 logical KV tokens (43.69× one maximum request) and 21.52–21.56 GiB KV per GPU. That large pool is concurrency capacity; it does not override the single-request correctness gate.

Measured throughput

Unique-prefix prefill, one request, no prefix reuse:

Table with columns: prompt, client-observed tok/s, server-accounted tok/s
prompt	client-observed tok/s	server-accounted tok/s
8K	2,983	—
32K	4,322	5,238
64K	4,637	5,326
128K	4,907	5,326

Aggregate target-only decode (MTP_TOKENS=0):

Table with columns: input context, C1 tok/s, C4 tok/s, C8 tok/s
input context	C1 tok/s	C4 tok/s	C8 tok/s
0	75.15	241.31	397.19
32K	69.51	234.64	349.38
128K	64.72	223.01	323.64

No preemption was observed in the qualification matrix. These are measurements from one 4× RTX PRO 6000 Blackwell PCIe host, not guarantees for other topology, clock, thermal, driver, storage, or request mixes.

Why K6 is the production default

On panel25's streaming lane, K6 records 0.013715 and K8 0.012384 nats. The sealed K6 value, 0.013723, belongs to a different lane; the former 0.001339 cross-lane contrast is not a same-lane fidelity improvement. K8 grows from 254 GB to 331 GB: about 77 GB / 30% more checkpoint bytes. Its qualified eager profile uses 78.94–78.97 GiB/GPU for weights plus non-torch allocations and leaves 7.10–7.14 GiB/GPU for KV; K6 leaves 21.52–21.56 GiB/GPU. On the recorded deployment matrices, K6 is 5.3–7.3× faster at short context, or 8.3–24.4× faster when the measured 32K/128K prefill cost is included. K6 is the appliance's production default for that measured resource/throughput tradeoff on 4×96 GiB. The checkpoint KL values do not establish that K8's native-serving output is closer to the teacher.

Docker Compose

Prerequisites: Linux x86-64, four visible RTX PRO 6000 Blackwell GPUs, NVIDIA driver ≥ 590.48.01 / CUDA 13.2 compatibility, the NVIDIA Container Toolkit, and roughly 300 GiB free persistent storage for the checkpoint plus caches. PCIe P2P on this card family requires NVIDIA's open kernel modules; see the RTX 6000 Pro multi-GPU notes.

yaml
name: glm53-k6
services:
  api:
    image: ghcr.io/malaiwah/glm52-exl3-vast@sha256:5a0d4b370e9f6a2ef85fa8b8c213492122554b34ba18d630a3a78130758914cf
    pull_policy: always
    restart: unless-stopped
    network_mode: host
    ipc: host
    shm_size: 32gb
    stop_grace_period: 2m
    ulimits:
      memlock:
        soft: -1
        hard: -1
    environment:
      MODEL_PROFILE: glm53-k6
      AUTH: key
      VLLM_API_KEY: ${VLLM_API_KEY:?set VLLM_API_KEY to a long random secret}
      HF_TOKEN: ${HF_TOKEN:-}
      SSH_ENABLED: "0"
      SOUL_ENABLED: "0"
      VERIFY_HEALTH_TIMEOUT_S: "3600"
    volumes:
      - /srv/glm53-turnkey:/workspace
      - /srv/glm53-cache:/cache
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 4
              capabilities: [gpu]

bash
sudo mkdir -p /srv/glm53-turnkey /srv/glm53-cache
export VLLM_API_KEY="$(openssl rand -hex 32)"
docker compose up -d
docker compose logs -f

First boot downloads about 237 GiB and can take substantial time. The container is ready only after the log reports

markdown

>>> Verified: serving; long-context retrieval verified

. API: http://HOST:8000/v1; dashboard: http://HOST:1111. The served model name is GLM-5.3-Flash-K6.

bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Authorization: Bearer $VLLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"GLM-5.3-Flash-K6","messages":[{"role":"user","content":"Reply with exactly READY"}],"max_tokens":256}'

Do not replace only the checkpoint path in another vLLM command. The profile, parent digest, runtime overlays, quantization, attention backend, DCP topology, KV calibration, scheduler, and graph widths are one qualified contract.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

malaiwah

Model Tree

Base

zai-org/GLM-5.3-Flash-BF16

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

QFS size–fidelity evidence

Interactive highlighted plot · PNG · SVG · CSV and exclusions · Full provenance JSON · Live cached image

Fidelity — sealed-ep8, five-cold-run qualification

⚠ Scope disclosure — this number is a panel25 number

Added 2026-08-29. Nothing here is a correction: 0.013723 is and remains the correct mean over the full 25-window panel. What changed is that the panel is now known to contain calibration-adjacent windows, so the scope has to travel with the number.

brandonmusic ran a 13-gram overlap scan of his sealed panel against its own calibration-role windows and found that the whole axis4_reasoning domain shares 37–39 % of its 13-grams with calibration material — despite the panel being clean at the document-hash level. Document-hash dedup is not enough. He excluded that domain and scored his primary numbers on the 17 windows that survive. The finding, the scan and the 0.05 threshold are his.

Every malaiwah number on this panel used all 25 windows, so every one of them carries the same contamination. Recomputed on his clean scope, from our own published per-window arrays (no GPU, no re-measurement — this is arithmetic on data already published):

Table with columns: panel25 (published), clean17 (his scope), move
panel25 (published) clean17 (his scope) move

The following raw values are a mixed-design inventory, not a ranking. Equal panel identity is necessary; the actual pair predicate must also pass.

Table with columns: Model, Mean KLD (nats), Size, Scope
Model	Mean KLD (nats)	Size	Scope
This K6 (sealed)	0.013723	254 GB	full panel × 5 bitwise-identical runs
This K6, streaming lane	0.013715	254 GB	full panel × 2 bitwise-identical runs; −8.5e-6 vs sealed (receipt)
K8 sibling	0.012384	331 GB

Historical single-window diagnostics, not comparable to the panel means:

Table with columns: Model, Mean KLD (nats), Size, Scope
Model	Mean KLD (nats)	Size	Scope
Official FP8 (his stack, v44)	0.024629	328 GB	1 window × 5 runs
NVFP4 (his stack, v44)	0.060535	~180 GB	1 window × 5 runs

Excess over control (descriptive subtraction)

Table with columns: panel KLD, excess over control
	panel KLD	excess over control
BF16 (floor)	0.011506	—
K8 (331 GB)	0.012384	0.000878
K6 (254 GB)	0.013715	0.002209

What this is (and is not)

Codec: EXL3-format TR3/MCG trellis (turboderp's exllamav3 kernels @ c5d9c657 did the encoding math), through brandonmusic's GLM-5.3 quantization pipeline with a small disclosed patch series.
Serving runtime: use malaiwah/glm52-exl3-vast with MODEL_PROFILE=glm53-k6. The image pins the qualified Glm5Next vLLM, B12X, EXL3, CUDA, and 21-file fail-closed runtime overlay as one contract.
Not stock exllamav3/TabbyAPI or stock upstream vLLM: those stacks do not carry this complete glm5_next + TR3/MCG K6 serving path.
Topology-neutral checkpoint: canonical unsharded tensors; TP layout is a load-time decision. The qualified deployment is exactly TP4/DCP4 on four 96 GiB RTX PRO 6000 Blackwell GPUs.
Measured memory: the packaged image loads 58.96 GiB of model tensors per rank. At GMU 0.93, final profiling reported 63.74–63.78 GiB weights + non-torch, 3.02 GiB peak activations, 0.45–0.46 GiB CUDA graphs, and 21.52–21.56 GiB KV per GPU.

Provenance & disclosed deviations

Family

Table with columns: K4, K6 (this), K8
	K4	K6 (this)	K8
Repo	brandonmusic's 4bpw	here	malaiwah/GLM-5.3-Flash-TR3-8bpw
Size	176 GB	254 GB	331 GB
Mean KLD (same panel)	0.024555	0.013723	0.012384

Lineage on the Hub

Related work on the same model, all measured on one panel in the quant-fidelity registry: brandonmusic 4bpw, 0xSero Dione Q4, orcarouter MLX. Collection: GLM-5.3-Flash — measured quants & fidelity.

Credits

Serving — live-qualified turnkey profile

Qualification result

Appliance source commit: a0d05f76994cf44f3667c0d2910d3b0e4d305d23
Qualified parent: verdictai/glm53-flash-exl3-k4@sha256:0f1cdcc8891f1cc3a444121eb61d366289a1cbba285f0892dcbb24bc94961692
Published appliance: ghcr.io/malaiwah/glm52-exl3-vast@sha256:5a0d4b370e9f6a2ef85fa8b8c213492122554b34ba18d630a3a78130758914cf
Shape: TP4 / DCP4 A2A, B12X sparse MLA, Triton MoE, EXL3 K6, calibrated NVFP4-DS MLA KV, MTP off, batch 3,072, C8, GMU 0.93
Request limit: 458,752 tokens; text-only qualification scope

Table with columns: document tokens, independent seed, retrieval, elapsed
document tokens	independent seed	retrieval	elapsed
384,612	20260831	3/3	81.058 s
449,462	20260901	3/3	95.839 s
449,461	20260903	3/3	94.578 s

Measured throughput

Unique-prefix prefill, one request, no prefix reuse:

Table with columns: prompt, client-observed tok/s, server-accounted tok/s
prompt	client-observed tok/s	server-accounted tok/s
8K	2,983	—
32K	4,322	5,238
64K	4,637	5,326
128K	4,907	5,326

Aggregate target-only decode (MTP_TOKENS=0):

Table with columns: input context, C1 tok/s, C4 tok/s, C8 tok/s
input context	C1 tok/s	C4 tok/s	C8 tok/s
0	75.15	241.31	397.19
32K	69.51	234.64	349.38
128K	64.72	223.01	323.64

Why K6 is the production default

Docker Compose

yaml
name: glm53-k6
services:
  api:
    image: ghcr.io/malaiwah/glm52-exl3-vast@sha256:5a0d4b370e9f6a2ef85fa8b8c213492122554b34ba18d630a3a78130758914cf
    pull_policy: always
    restart: unless-stopped
    network_mode: host
    ipc: host
    shm_size: 32gb
    stop_grace_period: 2m
    ulimits:
      memlock:
        soft: -1
        hard: -1
    environment:
      MODEL_PROFILE: glm53-k6
      AUTH: key
      VLLM_API_KEY: ${VLLM_API_KEY:?set VLLM_API_KEY to a long random secret}
      HF_TOKEN: ${HF_TOKEN:-}
      SSH_ENABLED: "0"
      SOUL_ENABLED: "0"
      VERIFY_HEALTH_TIMEOUT_S: "3600"
    volumes:
      - /srv/glm53-turnkey:/workspace
      - /srv/glm53-cache:/cache
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 4
              capabilities: [gpu]

bash
sudo mkdir -p /srv/glm53-turnkey /srv/glm53-cache
export VLLM_API_KEY="$(openssl rand -hex 32)"
docker compose up -d
docker compose logs -f

First boot downloads about 237 GiB and can take substantial time. The container is ready only after the log reports

markdown

>>> Verified: serving; long-context retrieval verified

. API: http://HOST:8000/v1; dashboard: http://HOST:1111. The served model name is GLM-5.3-Flash-K6.

bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Authorization: Bearer $VLLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"GLM-5.3-Flash-K6","messages":[{"role":"user","content":"Reply with exactly READY"}],"max_tokens":256}'

GLM-5.3-Flash-TR3-6bpw

README

QFS size–fidelity evidence

Fidelity — sealed-ep8, five-cold-run qualification

⚠ Scope disclosure — this number is a panel25 number

Excess over control (descriptive subtraction)

What this is (and is not)

Provenance & disclosed deviations

Family

Lineage on the Hub

Credits

Serving — live-qualified turnkey profile

Qualification result

Measured throughput

Why K6 is the production default

Docker Compose

Explore FriendliAI today

README

QFS size–fidelity evidence

Fidelity — sealed-ep8, five-cold-run qualification

⚠ Scope disclosure — this number is a panel25 number

Excess over control (descriptive subtraction)

What this is (and is not)

Provenance & disclosed deviations

Family

Lineage on the Hub

Credits

Serving — live-qualified turnkey profile

Qualification result

Measured throughput

Why K6 is the production default

Docker Compose