GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
crucible-labs
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
crucible-labs
Model Tree
Qwen/Qwen3.6-35B-A3B
ornith-ai/Ornith-1.0-35B
Input Modalities
Output Modalities
Supported Functionality
skinnyctax/Ornith-1.0-35B-Q6_K-Frankenstein-MTP-GGUF did this graft first, in the GGUF world — with a surgery script, donor-head-only files, and reported acceptance of 74.3% (Q6_K) to 79.5% (Q4_K_M) at --spec-draft-n-max 4. If you serve with llama.cpp, use theirs, not this.
This repo is the counterpart for the other stack:
It also makes the checkpoint self-consistent: Ornith's config.json declares mtp_num_hidden_layers: 1 while shipping no mtp.* tensors, and that mismatch breaks some quantization toolchains outright (discussion #15). Here the config and the weights finally agree.
If you already have Ornith-1.0-35B downloaded, you do not need this repo — you can build it locally in seconds with graft_mtp_head.py and save the 67 GB transfer. This repo exists for convenience.
The MTP head predicts token t+2 from the main model's hidden state. Ornith is a light post-train of Qwen3.6-35B-A3B, so that hidden space barely moved and the base's head still applies. Measured drift between Ornith and its base:
The result is self-verifying: a head that no longer matched the hidden space would produce drafts that get rejected, showing up immediately as low acceptance and a slower model. Instead acceptance is 70.3%, which is high enough to pay for the verify batch several times over.
(For loose context, the base model running its own MTP head measured 79% acceptance at n=2 in a separate test — but that was on llama.cpp rather than vLLM, so the two are not a controlled comparison and the difference should not be read as the cost of the post-train.)
Batch 1, temperature 0, warmup run discarded. All arms used identical serving flags apart from the speculative config; the control arm ran with speculation disabled. Single Strix Halo (gfx1151 / Radeon 8060S, 124 GB unified memory), ROCm 7.14, vLLM 0.27.1, TP=1.
| control | n=2 | n=4 | |
|---|---|---|---|
| prose tok/s | 27.83 | 38.60 (+38.7%) | 31.59 (+13.5%) |
| code tok/s | 27.50 | 42.51 (+54.6%) | 43.84 (+59.4%) |
| draft acceptance | — | 70.3% | 44.3% |
| tokens per forward | 1.000 | 2.393 | 2.753 |
Use num_speculative_tokens=2 unless you generate only code. n=4 drafts more but acceptance collapses to 44.3%, and prose ends up 22% worse than n=2. More speculation is not better speculation.
This hardware reads 5.488 GiB per token at a measured 224.2 GiB/s, which caps single-token decode at 40.9 tok/s no matter how well the kernels are tuned. At n=2, code generation measured 42.51 tok/s — past the ceiling.
That is the point of speculation rather than an error: when 2.4 tokens come out of one forward pass, they share a single weight read. Speculative decoding doesn't make memory faster — it amortizes one read across several tokens, which is the only way past a per-token bandwidth ceiling.
Forward cost is linear in verify rows (n+1 rows per forward). Fitting the three measured points above gives ~12.9 ms per row + ~23.3 ms fixed, which reproduces all three within 0.5 ms. Roughly 7 ms of that fixed term is the draft head reading its own 1.573 GiB at the measured bandwidth. The linearity is why the sweet spot is small: each extra speculative token costs a full verify row whether or not it is accepted, so it only pays while acceptance stays high.
--max-model-len 262144 --gpu-memory-utilization 0.90 --max-num-seqs 64 on a 124 GB unified-memory device — expect a completely different absolute figure on a discrete GPU. On long-context agent work this trade may matter more than the decode win.num_speculative_tokens ≤ 4 on ROCm. The skinny-GEMM guard is 0 < n <= 5 and the verify batch is n + 1, so at n=5 every dense GEMM falls to a slow path. It is a cliff, not a gradient.mamba_cache_mode='align' — the MTP class raises NotImplementedError on 'all'. With prefix caching enabled this was already the mode in use here, so it cost nothing, but check yours.min_p and logit_bias silently stop working under speculative decoding in vLLM. It logs this once at startup and never again. If your clients send them, they are being ignored.Speculation being output-preserving only under exact arithmetic is a real risk for structured output — n-gram speculation on this same model produced a malformed tool call, with the function-name field containing get_weather(location='Boston') and empty arguments. So tool calling was validated on a real benchmark rather than a smoke test:
In that comparison MTP first appeared to break 10 problems, 8 of them "no tool call emitted" — which looks exactly like speculation damaging structured output.
It was the harness's token cap. A <tool_call> block truncated mid-stream parses to no tool call and empty content, which is byte-identical to a model that emitted nothing. Replaying those problems at double the budget: 8 of 10 had hit finish_reason: "length", and they needed 1082–1461 tokens where the cap allowed 1024.
The general mechanism: speculation changes the generation trajectory, so it changes output length, so a token cap that was previously sufficient can start clipping. The regression then gets blamed on speculation, because that is the variable you changed. Record finish_reason per sample in your eval harness — without it, a truncated tool call and a silent model are indistinguishable after the fact.
mtp.* tensors in Qwen/Qwen3.6-35B-A3B and copy their bytes into a new shard.model.safetensors.index.json to map the 19 new keys to the new shard.Tensor counts: Ornith 31,666 + 19 = 31,685. The grafted tensors keep the base's 3D fused expert layout (mtp.layers.0.mlp.experts.gate_up_proj as [256, 1024, 2048]) even though Ornith stores its own experts 2D per-expert. That mismatch is fine and is the detail that makes this possible: the MTP module is loaded by a different model class than the main model, so the two layouts never have to agree.
No config surgery was needed — mtp_num_hidden_layers: 1 was already present in Ornith's config, and vLLM auto-derives architectures: ["Qwen3_5MoeMTP"].
Script: graft_mtp_head.py. It works for any base/finetune pair where the base ships an MTP head and the finetune is a light post-train of it.
n is workload-dependent — prose and code disagreed here and will disagree differently for you.This checkpoint combines material under two licenses. Both are permissive and compatible; neither is modified by this release.
| component | source | license |
|---|---|---|
| the 16 model shards, tokenizer, config, chat template | ornith-ai/Ornith-1.0-35B | MIT |
model-mtp-00001-of-00001.safetensors (19 mtp.* tensors) | Qwen/Qwen3.6-35B-A3B | Apache-2.0 |
The Apache-2.0 license text is included as LICENSE.qwen3.6-apache-2.0, per its section 4(a). The mtp.* tensors are redistributed unmodified; the only change to the Ornith checkpoint is additive (19 new tensor entries in the index).
Ornith-1.0-35B declares MIT in its model card metadata. Its repository does not currently ship a LICENSE file, so there is no upstream copyright notice to reproduce here; attribution is given by direct link above. If the Ornith authors would like specific notice text carried in this repo, open a discussion and it will be added.
Neither upstream project endorses this repository.
| ms per forward | 35.98 | 62.35 | 87.50 |
| KV cache capacity | baseline | −21.8% | −23.8% |