brandonmusic

GLM-5.2-EXL3-TR3-3.0bpw

Quantization layout

Table with columns: Component, Storage
Component	Storage
Routed MoE experts, layers 3-77	EXL3 Trellis, TP4 rank-sliced, 3.0 bpw target
Dense MLP layers 0-2	BF16
Shared-expert MLPs	BF16
Attention and sparse indexer	BF16
Embeddings and LM head	BF16
Norms, router gates, and e-score correction bias	BF16/FP32 source precision
MTP layer 78	BF16

The calibration manifest is included as calibration_manifest.json. It records 12,228 owner-corpus samples across general, legal, coding/agentic, and reasoning/termination axes. The model was calibrated at TP8 and packed for TP4.

Supported runtime

The tested runtime is published at verdictai/glm52-exl3-sparkinfer:

text
verdictai/glm52-exl3-sparkinfer:v27-gg-v20-scopefix-archkey-vllm0c79e41-sie603f74-cu132-sm120a

The supplied scripts pin the immutable registry manifest:

text
verdictai/glm52-exl3-sparkinfer:v27-gg-v20-scopefix-archkey-vllm0c79e41-sie603f74-cu132-sm120a@sha256:61ad3e2dee80b8c98409e55ffd32f7b60d4fb3c28938b935152734142f030f4e

It pins:

Gilded Gnosis v20 canonical vLLM 6722c1d (dev/gilded-gnosis) + EXL3 Trellis (rebased PR #139)
Sparkinfer v20 canonical 1a88b389 (master) + EXL3 Trellis fused arm (rebased PR #49)
CUDA 13.2, PyTorch 2.12, CUTLASS DSL 4.6.0, FlashInfer 801d57a, NCCL 2.30.4, SM120a
EXL3 Trellis MoE, B12X sparse MLA, DCP A2A, and MTP speculative decoding
NVFP4 DeepSeek-MLA KV cache using calibrated outer scales

v20 changes (Gilded Gnosis v20 base)

This release rebases the EXL3 Trellis backend onto the Gilded Gnosis v20 canonical heads (vLLM 6722c1d, Sparkinfer 1a88b389). The v20 base supplies the upstreamed MTP/DCP correctness fixes — head-major cross-rank BMM (vLLM #147), MTP verifier-decode dispatch (vLLM #164), and graph-resource isolation (vLLM #149). On top of that base this image adds (vLLM PR #139 / Sparkinfer PR #49, both rebased onto v20):

EXL3 Trellis MoE backend for rank-sliced GLM/DeepSeek routed experts.
MTP tool-call + DSA-crash fix — the structured-output grammar advances from the authoritative step delta so tool calling engages under speculative decoding, and has_indexer is derived from index_k so MTP draft steps that skip top-k no longer trip the fused-norm-rope assertion.
Dual-plan Trellis prefill — prefill batches route through the planned Trellis MoE.
SM120 + B12X flattening fix — MTP next_n>2 uses the native (B, next_n) sparse-indexer path instead of the DeepGEMM next_n<=2 flatten fallback, which had corrupted MTP-3 code generation.

Validated on 4x RTX PRO 6000 Blackwell 96 GB (TP4/DCP4, MTP-3 greedy, NVFP4 KV, FULL_AND_PIECEWISE cudagraphs):

Code generation — the previously reported ~50% syntax-error rate under MTP-3 is eliminated; generated Python/HTML is 94–100% syntactically valid across runs. The rare remaining edge is fp8-KV/DCP floating-point nondeterminism, not the prior systematic corruption.
LAVD long-context retrieval (r10/c5, temp 0): 10/10 (6 exact, 4 near, 0 fail) — up from 8/10 (2 fails) on the prior base. The v20 canonical MTP fixes plus the flattening fix drive the fail count from 2 to 0.

Asynchronous scheduling remains disabled as a correctness guard for this DCP4/MTP path; do not enable it on this release.

Quick start

Install Docker Engine, Docker Compose v2, the NVIDIA Container Toolkit, and the Hugging Face CLI. Then download the model and start the server:

bash
hf download brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw \
  --local-dir "$HOME/models/GLM-5.2-EXL3-TR3-3.0bpw"

cd "$HOME/models/GLM-5.2-EXL3-TR3-3.0bpw"
chmod +x server.sh
./server.sh start
./server.sh logs

The OpenAI-compatible endpoint is available locally at http://localhost:8000/v1. Here, localhost always means the machine on which the downloader starts this model; it does not refer to the model publisher's machine.

bash
curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "GLM-5.2-EXL3-TR3-3.0bpw",
    "messages": [{"role": "user", "content": "What is 2 + 2?"}],
    "temperature": 0,
    "max_tokens": 128
  }'

Useful controls:

bash
./server.sh status
./server.sh logs
./server.sh restart
./server.sh stop

Direct Compose use

server.sh exports the defaults and invokes the included docker-compose.yml. To run Compose directly, set the model and cache paths:

bash
export MODEL_DIR="$HOME/models/GLM-5.2-EXL3-TR3-3.0bpw"
export CACHE_DIR="$HOME/.cache/glm52-exl3-sparkinfer"
docker compose pull
docker compose up -d
docker compose logs --tail 100 -f glm52

The defaults can be overridden without editing the files:

Table with columns: Variable, Default, Purpose
Variable	Default	Purpose
`MODEL_DIR`	Directory containing `server.sh`	Model mount
`CACHE_DIR`	`~/.cache/glm52-exl3-sparkinfer`	Persistent JIT cache
`PORT`	`8000`	Host API port

The tested GPU order intentionally keeps physical GPU 3 away from TP rank 3. On another host, set CUDA_VISIBLE_DEVICES=0,1,2,3 or use the order appropriate for that machine.

The supplied Compose file binds only to loopback by default, so it does not publish the API to the LAN or internet.

Runtime validation

The exact published image completed all 81 model-shard loads, EXL3 initialization, Sparkinfer PCIe collective initialization, NVFP4 KV allocation, and full plus piecewise CUDA graph capture. The v20 base has a larger runtime footprint, so the release preset allocates 262,144 logical KV-cache tokens: 1,024 blocks x 64 tokens x DCP4, with 65,536 tokens local to each DCP rank. The configured per-request context cap is also 262,144. A GPU 0 with no other resident processes supports a larger KV pool and context (raise NUM_GPU_BLOCKS_OVERRIDE / MAX_MODEL_LEN within the KV capacity reported at startup).

Quality was validated with the release serving configuration (MTP-3, TP4/DCP4, concurrency 5):

Table with columns: Evaluation, v20 result, Prior base
Evaluation	v20 result	Prior base
LAVD r10/c5	10/10 (6 exact, 4 near, 0 fail)	8/10 (2 fail)
Code generation (Python/HTML, temp 0)	94–100% valid	~50% (reported bug)

The LAVD fail count dropping 2 → 0 reflects the v20 canonical MTP fixes plus the SM120+B12X flattening fix. Exact/near split varies run to run (fp8-KV/DCP nondeterminism); the 0-fail result is the stable signal.

The following prefill, decode, and KLD tables are indicative reference measured on the prior base; the v20 image is validated for correctness by the results above.

Cold standalone prefill results:

Table with columns: Requested context, Prompt tokens, TTFT, Client tok/s, Server tok/s
Requested context	Prompt tokens	TTFT	Client tok/s	Server tok/s
8K	8,201	5.46 s	1,502	1,507
64K	64,512	51.64 s	1,249	1,252
128K	128,881	109.00 s	1,182	1,185

Sustained decode used 20-second steady-state cells after warmup, zero input context, and continuous OpenAI stream usage:

Table with columns: Concurrency, 1, 2, 3, 4, 5, 6, 7, 8
Concurrency	1	2	3	4	5	6	7	8
Aggregate tok/s	48.9	112.9	154.0	188.2	218.2	239.4	253.9	266.8
Per-request tok/s	48.9	56.4

A separate 30-second C1 run measured 48.5 tok/s. No decode cell was underfilled, capacity-limited, or errored.

Five-run, 2,047-position DCP4 KLD against the same verified BF16 reference:

Table with columns: KV cache, Mean KLD, Sample SD, Min, Max
KV cache	Mean KLD	Sample SD	Min	Max
NVFP4 DeepSeek MLA	0.1124021	0.0025948	0.1086084	0.1156108
FP8	0.1036666	0.0018374	0.1016535	0.1066077

Full methodology, raw JSON, and copyable Rich TUI logs are in benchmarks/2026-07-22.

Source

License

The model and this derivative are released under the MIT license. See LICENSE and the upstream model card for attribution and usage terms.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

brandonmusic

Model Tree

Base

zai-org/GLM-5.2

Quantized

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

Quantization layout

Table with columns: Component, Storage
Component	Storage
Routed MoE experts, layers 3-77	EXL3 Trellis, TP4 rank-sliced, 3.0 bpw target
Dense MLP layers 0-2	BF16
Shared-expert MLPs	BF16
Attention and sparse indexer	BF16
Embeddings and LM head	BF16
Norms, router gates, and e-score correction bias	BF16/FP32 source precision
MTP layer 78	BF16

Supported runtime

The tested runtime is published at verdictai/glm52-exl3-sparkinfer:

text
verdictai/glm52-exl3-sparkinfer:v27-gg-v20-scopefix-archkey-vllm0c79e41-sie603f74-cu132-sm120a

The supplied scripts pin the immutable registry manifest:

text
verdictai/glm52-exl3-sparkinfer:v27-gg-v20-scopefix-archkey-vllm0c79e41-sie603f74-cu132-sm120a@sha256:61ad3e2dee80b8c98409e55ffd32f7b60d4fb3c28938b935152734142f030f4e

It pins:

Gilded Gnosis v20 canonical vLLM 6722c1d (dev/gilded-gnosis) + EXL3 Trellis (rebased PR #139)
Sparkinfer v20 canonical 1a88b389 (master) + EXL3 Trellis fused arm (rebased PR #49)
CUDA 13.2, PyTorch 2.12, CUTLASS DSL 4.6.0, FlashInfer 801d57a, NCCL 2.30.4, SM120a
EXL3 Trellis MoE, B12X sparse MLA, DCP A2A, and MTP speculative decoding
NVFP4 DeepSeek-MLA KV cache using calibrated outer scales

v20 changes (Gilded Gnosis v20 base)

EXL3 Trellis MoE backend for rank-sliced GLM/DeepSeek routed experts.
MTP tool-call + DSA-crash fix — the structured-output grammar advances from the authoritative step delta so tool calling engages under speculative decoding, and has_indexer is derived from index_k so MTP draft steps that skip top-k no longer trip the fused-norm-rope assertion.
Dual-plan Trellis prefill — prefill batches route through the planned Trellis MoE.
SM120 + B12X flattening fix — MTP next_n>2 uses the native (B, next_n) sparse-indexer path instead of the DeepGEMM next_n<=2 flatten fallback, which had corrupted MTP-3 code generation.

Validated on 4x RTX PRO 6000 Blackwell 96 GB (TP4/DCP4, MTP-3 greedy, NVFP4 KV, FULL_AND_PIECEWISE cudagraphs):

Code generation — the previously reported ~50% syntax-error rate under MTP-3 is eliminated; generated Python/HTML is 94–100% syntactically valid across runs. The rare remaining edge is fp8-KV/DCP floating-point nondeterminism, not the prior systematic corruption.
LAVD long-context retrieval (r10/c5, temp 0): 10/10 (6 exact, 4 near, 0 fail) — up from 8/10 (2 fails) on the prior base. The v20 canonical MTP fixes plus the flattening fix drive the fail count from 2 to 0.

Asynchronous scheduling remains disabled as a correctness guard for this DCP4/MTP path; do not enable it on this release.

Quick start

Install Docker Engine, Docker Compose v2, the NVIDIA Container Toolkit, and the Hugging Face CLI. Then download the model and start the server:

bash
hf download brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw \
  --local-dir "$HOME/models/GLM-5.2-EXL3-TR3-3.0bpw"

cd "$HOME/models/GLM-5.2-EXL3-TR3-3.0bpw"
chmod +x server.sh
./server.sh start
./server.sh logs

bash
curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "GLM-5.2-EXL3-TR3-3.0bpw",
    "messages": [{"role": "user", "content": "What is 2 + 2?"}],
    "temperature": 0,
    "max_tokens": 128
  }'

Useful controls:

bash
./server.sh status
./server.sh logs
./server.sh restart
./server.sh stop

Direct Compose use

server.sh exports the defaults and invokes the included docker-compose.yml. To run Compose directly, set the model and cache paths:

bash
export MODEL_DIR="$HOME/models/GLM-5.2-EXL3-TR3-3.0bpw"
export CACHE_DIR="$HOME/.cache/glm52-exl3-sparkinfer"
docker compose pull
docker compose up -d
docker compose logs --tail 100 -f glm52

The defaults can be overridden without editing the files:

Table with columns: Variable, Default, Purpose
Variable	Default	Purpose
`MODEL_DIR`	Directory containing `server.sh`	Model mount
`CACHE_DIR`	`~/.cache/glm52-exl3-sparkinfer`	Persistent JIT cache
`PORT`	`8000`	Host API port

The tested GPU order intentionally keeps physical GPU 3 away from TP rank 3. On another host, set CUDA_VISIBLE_DEVICES=0,1,2,3 or use the order appropriate for that machine.

The supplied Compose file binds only to loopback by default, so it does not publish the API to the LAN or internet.

Runtime validation

Quality was validated with the release serving configuration (MTP-3, TP4/DCP4, concurrency 5):

Table with columns: Evaluation, v20 result, Prior base
Evaluation	v20 result	Prior base
LAVD r10/c5	10/10 (6 exact, 4 near, 0 fail)	8/10 (2 fail)
Code generation (Python/HTML, temp 0)	94–100% valid	~50% (reported bug)

The following prefill, decode, and KLD tables are indicative reference measured on the prior base; the v20 image is validated for correctness by the results above.

Cold standalone prefill results:

Table with columns: Requested context, Prompt tokens, TTFT, Client tok/s, Server tok/s
Requested context	Prompt tokens	TTFT	Client tok/s	Server tok/s
8K	8,201	5.46 s	1,502	1,507
64K	64,512	51.64 s	1,249	1,252
128K	128,881	109.00 s	1,182	1,185

Sustained decode used 20-second steady-state cells after warmup, zero input context, and continuous OpenAI stream usage:

Table with columns: Concurrency, 1, 2, 3, 4, 5, 6, 7, 8
Concurrency	1	2	3	4	5	6	7	8
Aggregate tok/s	48.9	112.9	154.0	188.2	218.2	239.4	253.9	266.8
Per-request tok/s	48.9	56.4

A separate 30-second C1 run measured 48.5 tok/s. No decode cell was underfilled, capacity-limited, or errored.

Five-run, 2,047-position DCP4 KLD against the same verified BF16 reference:

Table with columns: KV cache, Mean KLD, Sample SD, Min, Max
KV cache	Mean KLD	Sample SD	Min	Max
NVFP4 DeepSeek MLA	0.1124021	0.0025948	0.1086084	0.1156108
FP8	0.1036666	0.0018374	0.1016535	0.1066077

Full methodology, raw JSON, and copyable Rich TUI logs are in benchmarks/2026-07-22.

Source

License

The model and this derivative are released under the MIT license. See LICENSE and the upstream model card for attribution and usage terms.

GLM-5.2-EXL3-TR3-3.0bpw

README

Quantization layout

Supported runtime

v20 changes (Gilded Gnosis v20 base)

Quick start

Direct Compose use

Runtime validation

Source

License

Explore FriendliAI today

README

Quantization layout

Supported runtime

v20 changes (Gilded Gnosis v20 base)

Quick start

Direct Compose use

Runtime validation

Source

License