Contents
Training lineage
- Expert pruning (REAP-style). Each of the 42 MoE layers of GLM-5.3-Flash (FP8) kept the 8 of its 288 routed experts with the highest router-weighted output norm (gate × ‖expert output‖), measured on a 102k-token calibration set. Router rows were cut to match. The shared expert, attention, the 3 dense layers, the MTP head and the vision encoder are unchanged. Held-out loss right after pruning: 3.725.
- Healing, stage 1. LoRA (rank 64, alpha 128, on every attention and MLP projection, routed experts included) plus fully trained routers. AdamW, learning rate 1e-4, warmup and cosine decay, 2,048-token chunks, loss on assistant turns only. 174 steps, 10.9M tokens. Held-out loss: 3.725 → 2.105.
- Healing, stage 2. Continued from stage 1 with a fresh optimizer on 12.1M new tokens (no conversation from stage 1 repeated), learning rate 8e-5, 193 steps. Held-out loss on a new held-out set: 2.449 → 2.218. This is the checkpoint released here.
The two held-out sets differ, so their losses are not comparable with each other.
Datasets
In total: 13 healing datasets and 3 calibration datasets (one used for both). Every dataset that shaped these weights is listed here and in this page's metadata. Healing data is only text written by GLM-5.3 or GLM-5.3-Flash, with one exception noted below. Training data is not re-released.
Healing
Table with columns: Dataset, What it is, Stage 1, Stage 2| Dataset | What it is | Stage 1 | Stage 2 |
|---|
| Zek-Takai/glm53-flash-harvest | Prose, chat, multi-turn, reasoning, code, structured output, agentic (GLM-5.3-Flash) | 65% | 69% |
| shanjiaz/glm-5.3-flash-open-perfectblend-regen-25pct | Mixed instructions | 15% | 2% |
| best-distill/glm-5.3-flash-distillation-chat | Instructions with reasoning | 7% | 3% |
| open-athena/GLM-5.3-RLVR1-Training-Rollouts-2026.09.25 | Mixed tasks (GLM-5.3) | 5% | 10% |
| sollamon/ox-alpha-glm-5.3-flash-distillation-coding-17k-raw | Coding | 3% | 3% |
| open-athena/GLM-5.3-tmax-swesmith-k8-traces-2026.10 | Terminal and bug-fix agent runs, passed only (GLM-5.3) | 1% | 7% |
| liangzhidanta/claude-code-glm53-swesmith-trajectories and its compact-aware version | Coding-agent runs, some with context compaction (GLM-5.3) | – | 3% |
| kaivoss/UltraData-SFT-Agent-2609-GLM53-20k | Agent runs in GLM-5.3's tool format. Exception: the text was written by the upstream models, not GLM | 2% | – |
| Rallex3, closestfriend, FranckAbgrall, flaviojoshua, Offlin33er function-calling sets | Tool calls (GLM-5.3-Flash) | 2% | 3% |
Shares are of each stage's tokens. Stage 1 drew 10.9M tokens from a 40M-token mix; stage 2 used 12.1M tokens from an 18M-token mix (the time budget ended the run first). In both stages, conversations where the model refused as an AI were dropped, and any sexual content involving minors was removed.
Calibration for pruning (stage 1 only)
These decided which experts were kept. The model was not trained on them.
Evaluation benchmarks
Architecture
Table with columns: Component, Value| Component | Value |
|---|
| Layers | 45 (3 dense + 42 MoE) |
| Width (hidden size) | 4096 |
| Experts | 8 routed + 1 shared per MoE layer, all 8 routed experts active on every token (from 288, top-8 of 288 in the base) |
| Expert hidden | 2048 |
| Attention | GLM-5.3-Flash hybrid sparse / linear attention (unchanged) |
| Extra heads | 1 MTP (multi-token prediction) layer, kept |
| Vision | GLM-5.3-Flash vision encoder (unchanged, not evaluated after pruning) |
| Vocabulary | 154,880 (GLM-5.3 tokenizer) |
| Context length | healed at 2,048 tokens; the base supports far more, but long context is untested after pruning |
| Parameters | 18.28B total: 16.06B layers, 1.27B embeddings and LM head, 0.39B MTP, 0.56B vision |
Because all 8 kept experts fire on every token, MP-02 runs at the speed of a dense 16B model, not a sparse MoE.
Evaluation
Scored with lm-evaluation-harness: zero-shot, the first 500 questions of each task, bf16 on one H100. The metric per task is the same as MP-01's harness: accuracy for ARC-Easy and WinoGrande, length-normalised accuracy for the rest. With 500 questions a task, differences under about 2 points are within noise. MP-01 was scored by its own harness on the same tasks, metrics and sample size, so treat small differences between the two as noise.
Comparison with MP-01
Table with columns: Rank, Model, Params, ARC-C, ARC-E, HellaSwag, OBQA, PIQA, SciQ, Wino, Avg (7)| Rank | Model | Params | ARC-C | ARC-E | HellaSwag | OBQA | PIQA | SciQ | Wino | Avg (7) |
|---|
| 1 | MP-02 16B Preview (this model) | 18.28B | 27.0 | 48.4 | 47.0 | 29.2 | 63.2 | 89.0 | 55.2 | 51.29 |
| 2 | MP-01 100M Preview | 100.6M | 29.4 | 56.4 | 31.8 | 30.2 | 64.4 | 73.0 | 53.8 | 48.43 |
Bold is the best score in each column.
Where this model stands
- 7-task average: about 3 points above MP-01, mostly from HellaSwag (+15.2) and SciQ (+16.0). It is behind MP-01 on ARC-Easy (−8.0) and slightly on ARC-Challenge, OpenBookQA and PIQA.
- That is a low result for its size. ARC-Challenge is near chance (25% for 4 choices). GLM-5.3-Flash itself was not run on this setup, but its published scores are far higher, so the pruning loss is large.
- These are multiple-choice log-likelihood tests. They don't measure chat, roleplay or thinking, which is what the healing focused on.
Progress across checkpoints of this model
Table with columns: Checkpoint, Held-out loss, ARC-C, ARC-E, HellaSwag, OBQA, PIQA, SciQ, Wino, Avg (7)| Checkpoint | Held-out loss | ARC-C | ARC-E | HellaSwag | OBQA | PIQA | SciQ | Wino | Avg (7) |
|---|
| Stage 1, step 174 | 2.105 (set A) | 27.0 | 48.0 | 47.2 | 27.4 | 64.4 | 89.4 | 56.8 | 51.46 |
| Stage 2, step 193 (this release) | 2.218 (set B) | 27.0 | 48.4 | 47.0 | 29.2 | 63.2 | 89.0 | 55.2 | 51.29 |
Stage 2 left the benchmarks where they were (all changes are within noise) and lowered loss on its own held-out set from 2.449 to 2.218.
Compute
Table with columns: Stage, Hardware, GPU hours, Tokens| Stage | Hardware | GPU hours | Tokens |
|---|
| Expert selection and pruning | Local consumer GPU (8 GB) | about 1.9 | 102k (calibration) |
| Healing, stage 1 (3 runs) | Cloud B200 | 2.85 | 10.9M |
| Healing, stage 2 (this release) | Cloud B200 | 2.60 | 12.1M |
| Benchmarks | Cloud H100 | 0.12 per checkpoint | |
Hours are wall-clock GPU time from the run logs, including model loading. Costs in credits are not listed.
Running it
llama.cpp (GGUF)
hf download Manprot/MP-02-16B-Preview MP-02-16B-Preview-Q4_K_M.gguf --local-dir MP-02llama-server -m MP-02/MP-02-16B-Preview-Q4_K_M.gguf -ngl 99 --n-cpu-moe 99 -c 8192
Needs a llama.cpp build with glm5-next support. On a desktop with an 8 GB GPU and DDR5 memory, keeping the experts on the CPU (--n-cpu-moe 99), it runs at about 6–7 tokens/s. On a single 24 GB GPU with everything on the GPU, it runs at about 28 tokens/s. For images, add --mmproj mmproj-MP-02-16B-Preview-BF16.gguf.
import torchfrom transformers import AutoTokenizer, Glm5NextForConditionalGeneration tok = AutoTokenizer.from_pretrained("Manprot/MP-02-16B-Preview")model = Glm5NextForConditionalGeneration.from_pretrained("Manprot/MP-02-16B-Preview", dtype=torch.bfloat16, device_map="auto")msgs = [{"role": "user", "content": "Write a short poem about the sea."}]ids = tok.apply_chat_template(msgs, add_generation_prompt=True, reasoning_effort="low", return_tensors="pt").to(model.device)print(tok.decode(model.generate(ids, max_new_tokens=400)[0][ids.shape[1]:]))
Needs transformers 5.17 or newer and about 37 GB of GPU memory in bf16.
Thinking and the chat template
MP-02 uses GLM-5.3's chat template, which always starts a thinking block and, with no reasoning_effort, sends Reasoning Effort: Max. All healing data was rendered at Max, so by default it thinks long on everything, even simple requests. For normal chat, pass reasoning_effort: "low" (for example "chat_template_kwargs": {"reasoning_effort": "low"} on llama-server). A thinking on/off switch in a chat UI has no effect with this template.
Suggested sampling: temperature 0.7, top-p 0.8, repeat penalty 1.05.
Limitations
- Preview quality. Most of GLM-5.3-Flash's knowledge is gone. It invents facts, gets simple arithmetic wrong and can talk confidently about things it does not know.
- Over-thinking. It thinks at maximum effort by default (see above) and can spend its whole token budget thinking before it answers.
- Agent habits. Agent traces in the healing data sometimes leak into chat: invented commands, sandbox paths such as
/home/user or /testbed, and fake tool output.
- Loops. It can repeat a phrase until the token limit. A repeat penalty helps.
- Identity. It was trained on GLM's own conversations and may say it is GLM, made by Z.ai.
- Few refusals. AI refusals were removed from the healing data, so it follows most requests. It has not been checked for safety; filter outputs yourself.
- Untested after pruning: vision, tool calling, long context and languages other than English.
Repository contents
model-*.safetensors, model.safetensors.index.json: merged weights in bf16 (norms in fp32)
config.json, generation_config.json, tokenizer.json, tokenizer_config.json, chat_template.jinja: model configuration and tokenizer
MP-02-16B-Preview-Q4_K_M.gguf: 4-bit GGUF for llama.cpp (12.9 GB)
mmproj-MP-02-16B-Preview-BF16.gguf: vision projector for llama.cpp (optional)
eval_results.json: full evaluation output for this checkpoint
LICENSE: MIT license with Z.ai's copyright notice
manprot-logo.png: logo
Citation
Made by the Manprot team, based on GLM-5.3-Flash by Z.ai. All credit for the original architecture and weights goes to Z.ai.
@misc{manprot2026mp02, title = {MP-02 16B Preview}, author = {{Manprot team}}, year = {2026}, publisher = {Hugging Face}, howpublished = {\url{https://huggingface.co/Manprot/MP-02-16B-Preview}}}
The datasets listed above belong to their authors and keep their own licenses.