GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
malaiwah
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
malaiwah
Model Tree
willfalco/GLM-5.2-EXL3-TR3-3.36bpw
Input Modalities
Output Modalities
Supported Functionality
A depth-pruned + expert-pruned ("REAP"-style) SIQ frankenmodel of willfalco/GLM-5.2-EXL3-TR3-3.36bpw (a rank-sliced Trellis/EXL3 "SIQ" quant of zai-org/GLM-5.2):
num_hidden_layers=16, MoE layers 3–15, MTP layer dropped).fp8_ds_mla KV, CUDA graphs).The maintainers of the vLLM+SparkInfer ("gilded-gnosis") stack serve the real GLM-5.2 SIQ in production on 4×RTX PRO 6000. Stack development needs a model with the same architecture, loader path, and attention tuple that boots on one consumer card, so engine changes can be torture-tested without touching production. Legume covers, unpatched, on one sm_120 GPU:
glm_moe_dsa architecture with the DSA sparse indexer activefp8_ds_mla KV cache (production's real KV dtype) with CUDA graphsv1 of Legume picked experts by lowest quantization round-trip MSE. This public build picked them by what the router actually uses on a narrow domain — a corpus of open-source license texts (recited by the full production GLM-5.2; Apache-2.0 held out):
enable_return_routed_experts — real engine routing (sigmoid noaux_tc
e_score_correction_bias, actual top-8), captured per token per layer.tournament/selection.json. Headline: seats split almost evenly across
cohorts (14–20 each per layer) — v1's lowest-quantization-MSE experts
kept only ~26% of their seats. Quantization fidelity and domain routing
mass are nearly orthogonal.Selection, not training — no gradient ever flowed. The router bias and gate rows were subset to the winners; every expert payload is byte-identical to its (re-encoded) parent tier.
| build | expert rule | mean logprob | ppl |
|---|---|---|---|
| v1 (private) | lowest quant MSE | −13.4155 | 670,329 |
| v2 (this repo) | license tournament | −12.4721 | 260,962 |
+0.94 nats/token, a 2.57× perplexity reduction — from expert selection alone, zero training. Both are still far beyond broken (uniform-random over this vocab would score ≈ −11.9; a 16-of-79-layer model has no continuity of thought), but the tournament measurably pulled the wreckage toward the license domain.
First words at temperature 0, same eval config, from
markdown
"Licensed under the Apache License, Version 2.0 (the"
agyipiilogilogilogilog… loops (and, in its classic serving config,
the immortal losacylosloslosloslosloslos)<arg_value> or simply stops — statistically closer to the
target text, expressed as reverent silencefp8_ds_mla KV + CUDA graphs: full engine init, clean 1/2/5/8/9-token
small-prompt batteryforward_mha) is unimplemented for this tuple —
prefills ≤512 tokens per request in offline mode; the API server path is
unaffected in production configsEverything under tools/: glm_franken.py (plan/dequant-gate/re-encode/
assemble), legume_tournament.py (select/seal/merge),
legume_routed_capture.py, legume_apache_eval.py,
gen_license_corpus.py, legume_v2_campaign.sh (the full orchestration),
plus the byte-pinned base encoder. The sealed plan (fingerprinted) ships in
the checkpoint as franken_plan.json; per-expert provenance in
tier_bitmap.json (franken_src_experts).
No benchmark was harmed in the making of this model, because none could be run. Do not deploy. Do not fine-tune. Do not ask it for legal advice — * especially* not about licenses.