GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
Minachist
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Minachist
Model Tree
deepreinforce-ai/Ornith-1.0-35B
Input Modalities
Output Modalities
Supported Functionality
INT8 (per-output-channel, symmetric) is applied to the routed-expert MLPs (gate_up_proj, down_proj) and the full-attention projections. The following are kept at BF16:
embed_tokens, lm_head, the MoE router (mlp.gate), the shared expert (shared_expert), the linear-attention / gated-delta mixers (linear_attn), and the entire vision tower (visual).
In total ~30,760 / 31,181 linear modules are quantized. the rest stay BF16.
| Field | Value |
|---|---|
| Base model | deepreinforce-ai/Ornith-1.0-35B |
| Method | AutoRound (intel/auto-round) |
| Scheme | W8A16 |
| Bits | 8 |
| Group size | -1 (per-output-channel) |
| Symmetric | yes |
| Format | auto_round (gptq-style packing) |
| Unquantized layers | embed_tokens, lm_head, mlp.gate, shared_expert, linear_attn, visual |
| Calibration data | 25 % NeelNanda/pile-10k + 75 % codeparrot/github-code-clean |
| Calibration samples | 1024 (256 pile + 768 github-code) |
| Iterations | 1000 |
| Batch size | 8 |
| Sequence length | 2048 |
| GPU used for quant | 2× RTX 3090 |
Quality was verified by measuring the KL divergence of the next-token distribution against the original BF16 model, KL(P_bf16 ‖ Q_int8), over 131,072 tokens (128 passages × 1024 tokens from
NeelNanda/pile-10k, held out from calibration). Lower is better.
| Metric | Value |
|---|---|
| Mean KL | 0.00348 nats |
| Median KL | 0.00139 nats |
| 99th-percentile KL | 0.0321 nats |
Reverse KL KL(Q‖P) | 0.00354 nats |
| Top-1 agreement | 97.5 % |
This checkpoint additionally ships a MTP draft module (model-mtp.safetensors) for speculative decoding.
This module is not part of the official Ornith release. It was rebuilt and re-trained as described below.
The MTP module here was initialized from the mtp.* tensors of the original Qwen/Qwen3.5-35B-A3B checkpoint and then fine-tuned to match Ornith-1.0-35B's output distribution (self-distillation, see below).
This is an unofficial community artifact. It is not affiliated with or endorsed by DeepReinforce AI or the Qwen team.
theblackcat102/evol-codealpaca-v1, m-a-p/CodeFeedback-Filtered-Instruction, openai/gsm8k, meta-math/MetaMathQA, OpenAssistant/oasst1, CohereLabs/aya_dataset, kunishou/hh-rlhf-49k-ja, kunishou/oasst1-89k-ja
Single-stream decode on 2× RTX 3090 (TP=2), vLLM nightly (0.23.1rc1), this exact checkpoint:
| Configuration | Decode tok/s | Acceptance rate | Per-position acceptance | Mean accepted length |
|---|---|---|---|---|
| No speculative decoding | ≈ 166 | — | — | 1.0 |
+ MTP, num_speculative_tokens=3 | ≈ 267 | 0.66 | 0.836 / 0.649 / 0.496 | 2.98 |
MTP row: 160 held-out mixed instruction + code prompts × 512 tokens each.
Baseline row: 28 coding prompts × 1024 tokens each.
k=3 gave the best single-stream throughput in testing. k=2 yields a higher per-draft acceptance rate (≈ 0.70–0.75) at slightly lower throughput.
The MTP routed experts (mtp.layers.0.mlp.experts) are INT8-quantized with GPTQ, using the same AutoRound scheme as the backbone. Everything else in the MTP block stays in BF16.
vLLM is recommended (tested with vllm/vllm-openai:nightly, 0.23.1rc1).
To enable MTP speculative decoding:
bash
vllm serve <this-repo> \--tensor-parallel-size 2 \--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
For text-only serving on 2×24 GB, --language-model-only --mamba-cache-mode align were used in testing.
If you're having quality issues while using MTP, it's likely a vLLM issue since MTP shouldn't degrade the output at all.
For more information, please see: this reddit thread, and this vLLM pull