GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
malaiwah
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
malaiwah
Model Tree
willfalco/GLM-5.2-EXL3-TR3-3.36bpw
Input Modalities
Output Modalities
Supported Functionality
A 20-layer, 724-expert frankenmodel of GLM-5.2 (via willfalco's EXL3-TR3
3.36bpw SIQ quant): old layers 0–15 + 74–77, per-layer expert widths
[56, 52, 52, 48, 40, 48, 56, 48, 48, 36, 36, 36, 36, 28, 32, 36, 36]
(layers 3–19), experts selected by routed-count tournament on a license
corpus (prefix layers), MSE rank (mid layers), each re-encoded at its parent
K3/K4 tier at TP1. ~23.2 GiB; boots on one RTX 5090 with the DSA sparse
indexer active and fp8_ds_mla KV.
patches/)Stock vLLM/GG builds assume one expert count for all MoE layers. v3's heterogeneous widths need:
patches/deepseek_v2.py — honors n_routed_experts_per_layer (per-layer
width override via a config shallow-copy keyed on the layer prefix)patches/exl3.py — SIQ loader accepts a per-layer experts_per_layer
list (three sites: bitrate-map validation, uniform fallback, width
cross-check)Mount them over the originals (-v patch.py:$SITE/…:ro). Everything else —
DSA indexer, MLA, fp8_ds_mla KV, dynamic tiers — is bone stock. That is
the point: these ~40 patched lines are the entire distance between the
production stack and heterogeneous-width MoE serving.
| round | change | result | verdict |
|---|---|---|---|
| v1 | 16L/64E, lowest-MSE experts | −13.42 / 670k | baseline |
| v2 | + routing tournament | −12.47 / 261k | shipped as Legume The First |
| 1b | 17L (add old-12), asymmetric widths | −12.00 / 163k | accept |
| 2a | drop old-74, widen early | −13.77 / 954k | reject — tail is sacred |
Uniform-random over this vocab ≈ −11.93. v3 predicts held-out license text better than chance — from selection and arrangement alone, zero gradient steps. Perplexity fell 5.5× from v1.
What the search taught us about a lobotomized MoE:
fp8_ds_mla KV; eval config runs eager at
512-ctx (the 31.4 GiB card is a razor fit at this size)tournament/; the sealed plan
(fingerprint 6a20624f…) ships as franken_plan.jsontools/ contains the complete loop: legume_v3.py (pools / plans / merge /
assemble), legume_v3_round.sh (one surgery round), capture + eval scripts,
the corpus generator, and the byte-pinned base encoder. Every expert of every
candidate layer was encoded once into a 256-expert pool; every round is a
pure merge — the whole 12-round search re-encoded nothing.
The patient is now measurably better than a coin flip at licenses and worse than one at everything else. Do not deploy. Do not fine-tune. Do not ask it to review your LICENSE file — it will simply feel seen.
| 2b | 18L (add old-13) | −11.94 / 154k | accept — crossed the uniform-random line |
| 3a | 19L (add old-14) | −11.85 / 140k | accept |
| 4 | 21L (add old-15+16 at once) | −12.12 / 184k | reject — width starvation |
| 4b | 20L (add old-15 only) | −11.70 / 121k | accept — FINAL |
| 5 | 21L (add old-16, balanced trims) | −12.10 / 180k | reject — prefix saturated |
| 6 | shift 12 seats tail→early | −11.97 / 158k | reject — tail width also sacred |
| 7 | re-select mid layers by routing | −11.74 / 125k | reject |
| 8 | extend tail backward (old-73) | −13.16 / 517k | reject — the wound interface is load-bearing |