drowzeys

keys-GLM-5.3-EXL3-Abliterated

Abliteration (what was edited, and how)

Credit: the BF16 derisk projections are Blackfrost AI's (Blackfrost-Research, GLM-5.3-DERISKED-BF16). We adapted them to this EXL3 routed-expert quant with the MTP block and the DFlash2 draft checkpoints preserved.

The reference edit, measured tensor-by-tensor against zai-org/GLM-5.3-BF16, is a rank-1 orthogonal projection applied in bf16 before quantization:

W' = W − α · r (rᵀ W), α = 3.0, one unit direction r per layer, on self_attn.o_proj and every residual-writing down_proj (dense, shared experts, routed experts), layers 2–49 only (layers 0–1, 50–78 and the MTP block untouched).

What is baked into these shards (see ABLIT_BAKE_REPORT.json for the per-tensor relative change):

Table with columns: Tensor, Layers, Edit, Fidelity
Tensor	Layers	Edit	Fidelity
`self_attn.o_proj`	2–49	replaced by the exact DERISKED-BF16 tensor (cast to fp16)	exact
`mlp.down_proj` (dense)	2	projected with the recovered r₂, α=3.0	rank-1 fit residual 2–4%
`mlp.shared_experts.down_proj`	2–49	projected with r_L, α=3.0	same
`mlp.experts.*.down_proj` (routed)	2–49	not edited — these are the 3-bit EXL3 trellis tensors; editing them requires re-quantizing from DERISKED-BF16	—

The directions were recovered by SVD of (W_derisked − W_stock) per layer (s1/s2 ≈ 300–850, α fits 2.999–3.001 on every layer). A load-time variant of the same edit (Mia's ablit_runtime.py transplant + our down_proj projection hook) is in serving/; it produces the same weights this repo ships pre-applied, so serve this repo with ABLIT=0.

Coherence checks on the served model (greedy): factual/arithmetic, code, multi-step reasoning, long creative prose and benign "refusal-shaped" prompts all answered correctly and fully. Speed is unchanged versus the base build (MTP acceptance per step identical).

Preserved speculators: the bf16 MTP block (layer 78) is in the shards and serves as the MTP k=3 speculator; the full-model DFlash2 draft (GLM-5.3-DFlash2) is kept separately and works without DCP.

What this is: the complete zai-org/GLM-5.3 (78 layers, 256 routed experts, DeepSeek-style sparse MLA attention with indexer, one MTP block) with the routed experts quantized to 3-bit EXL3 trellis and everything else kept bf16/fp16. 308 GB in 41 safetensors shards. It serves on four NVIDIA DGX Spark (GB10) with TP=4 + decode-context-parallel 4 at a 1,000,000-token context, inside Mia AI Lab's vLLM container for the GLM-5.3-Flash EXL3 (ghcr.io/miaai-lab/glm-5.3-flash-2x-dgx-sparks:exl3) plus the small patch set shipped in serving/.

Table

Source	`zai-org/GLM-5.3` (FP8 block-scaled, 756 GB, 141 shards)
Quantizer	exllamav3 1.4.5 (`convert.py`, `-hb 16 -mb 16 -cpi 0`, calibration 250 rows × 2048)
Routed experts	57,600 matrices (75 MoE layers × 256 experts × gate/up/down) at 3.00 bpw, `out_scales: always`
Codebook	layer 3 = `mcg`, layers 4–77 = `mul1` (see Codebook note); the serving overlay handles both per layer
Everything else

Why we made it this way

Every previous EXL3 GLM-5.3 we tried to serve on DGX Spark cost hours-to-days of patching: the published quants were made for TabbyAPI/exllamav3 (full-model EXL3, 2-bit heads, mul1 everywhere) and vLLM has no full-model EXL3 — Mia's container only implements EXL3 for the routed experts, with attention/dense/head native. So this time we quantized to the container's ABI first:

scope = routed experts only (the only thing the overlay's Exl3Config quantizes), uniform integer bits in the overlay's allowed set {3,4,5,6} → 3-bit, because 4-bit experts (≈101 GB/rank) do not fit four 128 GB GB10s next to a 1M KV cache; 3-bit ≈ 78 GB/rank does.
-hb 16 -mb 16: head and MTP stay bf16 so the stock vLLM paths load them.
suffix layout trellis/suh/svh/<codebook marker> exactly as the overlay's loader expects, quantization_config.json with quant_method=exl3, bits=3, scope=glm53_routed_experts_only.

It still needed a handful of patches (all in serving/), but they are small, mechanical, and now documented — the point of this repo.

How it was made (4× DGX Spark "encode farm")

One 753B model does not fit a single 128 GB Spark for quantization, and exllamav3 has no multi-GPU convert. We built a rolling pipeline (quant/orchestrate.py):

Rolling source shards. The FP8 source is 756 GB and not layer-ordered; the orchestrator keeps a 5-shard download window (sha-checked), runs convert.py for two layers at a time in a fresh container (--max_module L+1, then -r resume), and after each step rewrites finished shards dropping only model.layers.L.mlp.experts.* (norms/attention stay — the compile step needs them). Peak disk ≈ 660 GB; permanent shards = embed/lm_head, final norm, and the layer-78 MTP shard (must exist at every resume).
Remote expert-encoding farm (quant/exl3_farm.py + quant/patch_farm.py). exllamav3's per-layer H (Hessian) capture is serial, but the 768 expert matrices per layer are independent once H is final. The patch adds remote work threads to quantize_linears_parallel: it ships each matrix + its finalized H over TCP (length-prefixed torch.save) to worker containers on the other three Sparks, which return trellis/suh/svh/marker. Verified bit-identical to local encoding. One MoE layer: 49 min single-GPU → ~20–22 min on four. Whole model: ~27 wall-clock hours including a power-cut restart.
A power interruption at 60/78 lost nothing: exllamav3 checkpoints after every layer; the in-flight layer is simply redone.

quant/recipe.yaml (58,167 tensor entries) is included; quant/build.sh + quant/patch_aarch64.py build exllamav3 1.4.5 on aarch64/CUDA 13 inside the Mia container image (AVX all-reduce/CPU-MoE stubs + pause intrinsic).

Codebook note (read before you quantize with exllamav3 ≥ 1.4)

convert.py -r (resume) lets the command-line default win over the stored args for the codebook (("codebook", True, "mul1") in the arg table). We passed -cb mcg on the first run only; every resumed 2-layer run silently quantized with mul1. Result: layer 3 is mcg, layers 4–77 are mul1. Quality is identical (both are 3-bit trellis codebooks), but Mia's overlay was mcg-only — which is why serving/exl3.py now detects the codebook per layer and drives the kernels' (mcg, mul1) flag pairs. If you resume an exllamav3 job, re-pass every quant flag.

Serving on 4× DGX Spark (what `serving/` contains and why)

Base: ghcr.io/miaai-lab/glm-5.3-flash-2x-dgx-sparks:exl3 + its start-tp4.sh launcher (github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks). Image layers we add (Dockerfiles included, build order dcp → mul1 → v145 → keys → keys3):

Table with columns: Patch, Why
Patch	Why
`exl3.py` (overlay)	+ `mul1` marker tensors and per-layer codebook detection; markers may be 0-d scalars; `NullConfig/InferParams` stub for exllamav3 ≥ 1.x's `LinearEXL3`; MTP block exempt from EXL3 so `SPEC_METHOD=mtp` loads the bf16 MTP experts through the stock path
exllamav3 1.4.5 swap (`Dockerfile.v145`)	the image's exllamav3 0.0.43 `exl3_moe` kernel hard-rejects `mul1` (`TORCH_CHECK`); 1.4.5 has the same 29-arg signature (+`num_active`) and accepts it. Built against the same base torch, so the drops in

Launcher-level gotchas we hit (all in env.tp4.example comments): --ulimit nofile=1048576 (NCCL "Too many open files" at 4 ranks), a rank whose fabric IP sits on the second CX7 port needs its own WORKER_CX7_IF/IB, the NFS server node must bind-mount its own export at the shared path, gpu_memory_utilization above ~0.82 is refused by vLLM's startup guard (the worker already holds ~14 GiB of CUDA/NCCL state at check time) — use --kv-cache-memory-bytes to size the KV cache explicitly.

Measured (4× GB10, TP=4, DCP=4, CUDA graphs, fp8 KV, MTP k=3)

Production config: MTP k=3 speculator (the checkpoint's own bf16 MTP block), 1M context, idle serve, 256-token outputs, temperature 0.6 / top_p 0.95, thinking off. tok/s is decode only (completion tokens ÷ (wall − TTFT)); aggregate is summed across streams.

Table with columns: Task, C1 per-stream, C2 agg, C4 agg, C8 agg, accepted tok/step
Task	C1 per-stream	C2 agg	C4 agg	C8 agg	accepted tok/step
prose	11.1	15.4	21.0	30.5	1.8
code	15.0	24.4	28.4	41.7	2.5
list	13.5	21.7

TTFT 0.6–2.0 s across C1–C8. Decode rises with depth as the fixed per-step DCP cost amortizes (8→12 tok/s from short context to 1M). Prefill ≈ 320–410 tok/s is the limiter at depth: a 979K prompt is ~52 min to first token. Speculative gain is acceptance-bound — code/list/essay accept 2.3–2.5 draft tokens/step, prose ~1.8 — so 25–30 tok/s single-stream is not reachable on prose with this MTP head.

DFlash2 draft (parked): the full-model DFlash2 draft (incoai/GLM-5.3-DFlash2) can be made to boot at DCP=4 (see serving/ patches: replicate the sliding-window KV per rank, --prefix-match-unit 16), but vLLM has no correct decode-context-parallel path for sliding-window attention, so it generates garbage. Full-model DFlash2 at DCP needs a DCP-aware SWA attention kernel (upstream), not a config change. MTP k=3 is the shipping speculator.

Quick start (4 Sparks)

bash
git clone https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks glm53-mia && cd glm53-mia
python3 serving/patch_tp4_launcher.py start-tp4.sh          # LOCAL_MODEL_DIR + ABLIT_HOST_DIR knobs
# build the image chain on EVERY rank (see serving/Dockerfile.* ; ~1 min each on top of the Mia base)
cp serving/env.tp4.example .env.tp4                          # set rank IPs / NIC pins / LOCAL_MODEL_DIR=<this repo on shared storage>
SKIP_PULL=1 SKIP_SHIP=1 SKIP_BUILD=1 SKIP_DOWNLOAD=1 SKIP_SYNC=1 SKIP_OVERLAY_VERIFY=1 ./start-tp4.sh

OpenAI-compatible API on :8888, served model name GLM-5.3-EXL3. Sampling: temperature 1.0, top_p 0.95, thinking on (GLM defaults).

Lessons for "quantize with the serving container in mind"

Read the serving overlay's loader first: allowed scope, bits set, tensor suffixes, marker dtype/shape, codebook — and quantize to that ABI, not to the quantizer's defaults.
Keep anything the container serves natively (attention, dense, head, MTP) unquantized; the memory cost is small next to the experts and it removes whole classes of patches.
Verify tensor suffixes after the first resumed step, not at the end (see the codebook note).
Budget memory per rank for weights + KV + the ~14 GiB the process holds before vLLM measures anything; size KV explicitly.
A per-layer encode farm turns a multi-day single-GPU convert into an overnight job on the same boxes that will serve the model.

Credits

Zhipu / zai-org for GLM-5.3 · turboderp for exllamav3 · Mia AI Lab for the GB10 EXL3 vLLM container and launcher · Blackfrost AI (Blackfrost-Research) for the BF16 derisk projections (GLM-5.3-DERISKED-BF16) that this build bakes into o_proj/down_proj, adapted by keys to EXL3 with the MTP/DFlash2 checkpoints preserved · Quantized and served by @drowzeys on a 4× DGX Spark cluster, 2026-09.

License: the weights inherit the zai-org/GLM-5.3 license (MIT). Patches in serving/ are MIT; they modify Apache-2.0 vLLM/SGLang/exllamav3 code — see their headers.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

drowzeys

Model Tree

Base

zai-org/GLM-5.3

Quantized

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

Abliteration (what was edited, and how)

The reference edit, measured tensor-by-tensor against zai-org/GLM-5.3-BF16, is a rank-1 orthogonal projection applied in bf16 before quantization:

What is baked into these shards (see ABLIT_BAKE_REPORT.json for the per-tensor relative change):

Table with columns: Tensor, Layers, Edit, Fidelity
Tensor	Layers	Edit	Fidelity
`self_attn.o_proj`	2–49	replaced by the exact DERISKED-BF16 tensor (cast to fp16)	exact
`mlp.down_proj` (dense)	2	projected with the recovered r₂, α=3.0	rank-1 fit residual 2–4%
`mlp.shared_experts.down_proj`	2–49	projected with r_L, α=3.0	same
`mlp.experts.*.down_proj` (routed)	2–49	not edited — these are the 3-bit EXL3 trellis tensors; editing them requires re-quantizing from DERISKED-BF16	—

Table

Source	`zai-org/GLM-5.3` (FP8 block-scaled, 756 GB, 141 shards)
Quantizer	exllamav3 1.4.5 (`convert.py`, `-hb 16 -mb 16 -cpi 0`, calibration 250 rows × 2048)
Routed experts	57,600 matrices (75 MoE layers × 256 experts × gate/up/down) at 3.00 bpw, `out_scales: always`
Codebook	layer 3 = `mcg`, layers 4–77 = `mul1` (see Codebook note); the serving overlay handles both per layer
Everything else

Why we made it this way

scope = routed experts only (the only thing the overlay's Exl3Config quantizes), uniform integer bits in the overlay's allowed set {3,4,5,6} → 3-bit, because 4-bit experts (≈101 GB/rank) do not fit four 128 GB GB10s next to a 1M KV cache; 3-bit ≈ 78 GB/rank does.
-hb 16 -mb 16: head and MTP stay bf16 so the stock vLLM paths load them.
suffix layout trellis/suh/svh/<codebook marker> exactly as the overlay's loader expects, quantization_config.json with quant_method=exl3, bits=3, scope=glm53_routed_experts_only.

It still needed a handful of patches (all in serving/), but they are small, mechanical, and now documented — the point of this repo.

How it was made (4× DGX Spark "encode farm")

One 753B model does not fit a single 128 GB Spark for quantization, and exllamav3 has no multi-GPU convert. We built a rolling pipeline (quant/orchestrate.py):

Rolling source shards. The FP8 source is 756 GB and not layer-ordered; the orchestrator keeps a 5-shard download window (sha-checked), runs convert.py for two layers at a time in a fresh container (--max_module L+1, then -r resume), and after each step rewrites finished shards dropping only model.layers.L.mlp.experts.* (norms/attention stay — the compile step needs them). Peak disk ≈ 660 GB; permanent shards = embed/lm_head, final norm, and the layer-78 MTP shard (must exist at every resume).
Remote expert-encoding farm (quant/exl3_farm.py + quant/patch_farm.py). exllamav3's per-layer H (Hessian) capture is serial, but the 768 expert matrices per layer are independent once H is final. The patch adds remote work threads to quantize_linears_parallel: it ships each matrix + its finalized H over TCP (length-prefixed torch.save) to worker containers on the other three Sparks, which return trellis/suh/svh/marker. Verified bit-identical to local encoding. One MoE layer: 49 min single-GPU → ~20–22 min on four. Whole model: ~27 wall-clock hours including a power-cut restart.
A power interruption at 60/78 lost nothing: exllamav3 checkpoints after every layer; the in-flight layer is simply redone.

Codebook note (read before you quantize with exllamav3 ≥ 1.4)

Serving on 4× DGX Spark (what `serving/` contains and why)

Table with columns: Patch, Why
Patch	Why
`exl3.py` (overlay)	+ `mul1` marker tensors and per-layer codebook detection; markers may be 0-d scalars; `NullConfig/InferParams` stub for exllamav3 ≥ 1.x's `LinearEXL3`; MTP block exempt from EXL3 so `SPEC_METHOD=mtp` loads the bf16 MTP experts through the stock path
exllamav3 1.4.5 swap (`Dockerfile.v145`)	the image's exllamav3 0.0.43 `exl3_moe` kernel hard-rejects `mul1` (`TORCH_CHECK`); 1.4.5 has the same 29-arg signature (+`num_active`) and accepts it. Built against the same base torch, so the drops in

Measured (4× GB10, TP=4, DCP=4, CUDA graphs, fp8 KV, MTP k=3)

Table with columns: Task, C1 per-stream, C2 agg, C4 agg, C8 agg, accepted tok/step
Task	C1 per-stream	C2 agg	C4 agg	C8 agg	accepted tok/step
prose	11.1	15.4	21.0	30.5	1.8
code	15.0	24.4	28.4	41.7	2.5
list	13.5	21.7

Quick start (4 Sparks)

bash
git clone https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks glm53-mia && cd glm53-mia
python3 serving/patch_tp4_launcher.py start-tp4.sh          # LOCAL_MODEL_DIR + ABLIT_HOST_DIR knobs
# build the image chain on EVERY rank (see serving/Dockerfile.* ; ~1 min each on top of the Mia base)
cp serving/env.tp4.example .env.tp4                          # set rank IPs / NIC pins / LOCAL_MODEL_DIR=<this repo on shared storage>
SKIP_PULL=1 SKIP_SHIP=1 SKIP_BUILD=1 SKIP_DOWNLOAD=1 SKIP_SYNC=1 SKIP_OVERLAY_VERIFY=1 ./start-tp4.sh

OpenAI-compatible API on :8888, served model name GLM-5.3-EXL3. Sampling: temperature 1.0, top_p 0.95, thinking on (GLM defaults).

Lessons for "quantize with the serving container in mind"

Read the serving overlay's loader first: allowed scope, bits set, tensor suffixes, marker dtype/shape, codebook — and quantize to that ABI, not to the quantizer's defaults.
Keep anything the container serves natively (attention, dense, head, MTP) unquantized; the memory cost is small next to the experts and it removes whole classes of patches.
Verify tensor suffixes after the first resumed step, not at the end (see the codebook note).
Budget memory per rank for weights + KV + the ~14 GiB the process holds before vLLM measures anything; size KV explicitly.
A per-layer encode farm turns a multi-day single-GPU convert into an overnight job on the same boxes that will serve the model.

Credits

License: the weights inherit the zai-org/GLM-5.3 license (MIT). Patches in serving/ are MIT; they modify Apache-2.0 vLLM/SGLang/exllamav3 code — see their headers.

keys-GLM-5.3-EXL3-Abliterated

README

Abliteration (what was edited, and how)

Why we made it this way

How it was made (4× DGX Spark "encode farm")

Codebook note (read before you quantize with exllamav3 ≥ 1.4)

Serving on 4× DGX Spark (what `serving/` contains and why)

Measured (4× GB10, TP=4, DCP=4, CUDA graphs, fp8 KV, MTP k=3)

Quick start (4 Sparks)

Lessons for "quantize with the serving container in mind"

Credits

Explore FriendliAI today

README

Abliteration (what was edited, and how)

Why we made it this way

How it was made (4× DGX Spark "encode farm")

Codebook note (read before you quantize with exllamav3 ≥ 1.4)

Serving on 4× DGX Spark (what `serving/` contains and why)

Measured (4× GB10, TP=4, DCP=4, CUDA graphs, fp8 KV, MTP k=3)

Quick start (4 Sparks)

Lessons for "quantize with the serving container in mind"

Credits

keys-GLM-5.3-EXL3-Abliterated

README

Abliteration (what was edited, and how)

Why we made it this way

How it was made (4× DGX Spark "encode farm")

Codebook note (read before you quantize with exllamav3 ≥ 1.4)

Serving on 4× DGX Spark (what serving/ contains and why)

Measured (4× GB10, TP=4, DCP=4, CUDA graphs, fp8 KV, MTP k=3)

Quick start (4 Sparks)

Lessons for "quantize with the serving container in mind"

Credits

Explore FriendliAI today

README

Abliteration (what was edited, and how)

Why we made it this way

How it was made (4× DGX Spark "encode farm")

Codebook note (read before you quantize with exllamav3 ≥ 1.4)

Serving on 4× DGX Spark (what serving/ contains and why)

Measured (4× GB10, TP=4, DCP=4, CUDA graphs, fp8 KV, MTP k=3)

Quick start (4 Sparks)

Lessons for "quantize with the serving container in mind"

Credits

Serving on 4× DGX Spark (what `serving/` contains and why)

Serving on 4× DGX Spark (what `serving/` contains and why)