The MTP finding
Ornith-1.5-35B-A3B ships 785 mtp.* tensors that are random initialisation, not trained weights: every projection has
std = 0.0200 with Gaussian kurtosis 3.0 (i.e. exactly initializer_range=0.02), and its norm weights sit near 0.02 instead
of ~1. Used as a speculative draft it accepts only ~13% of tokens (pure chance). This model replaces that placeholder with
the trained MTP head from Qwen3.6-35B-A3B, which transfers cleanly because Ornith's language tower is only ~0.2–1%
away from Qwen3.6 (measured cosine per tensor group).
Measured with llama.cpp speculative decoding (Q4_K_M main + Q8_0 draft, RTX 3090):
Table with columns: draft head, acceptance, mean accepted run| draft head | acceptance | mean accepted run |
|---|
| Qwen3.6 grafted (this model) | 0.55–0.75 (code high, chat lower) | 3.2–4.0 tokens |
| Ornith-1.5 stock (random init) | ~0.13 | ~1.5 |
Merge recipe
- TIES (density 0.25, λ=1.0), computed in fp32 over the Qwen3.6-35B-A3B base, on all text weights.
- MoE router gates: kept verbatim from Ornith (never averaged — routing is where naive MoE merges break).
- Vision tower: kept verbatim (bit-identical between Ornith and Qwen3.6 anyway; this model keeps Qwen3.6's multimodal eyes).
mtp.*: Qwen3.6's trained head, verbatim (fused-expert layout, loads with the same Qwen3_5MoeForConditionalGeneration class).
Evaluation (Q4_K_M, single RTX 3090, temp 0.1)
Table with columns: this merge, stock Ornith-1.5 | this merge | stock Ornith-1.5 |
|---|
| 15-task Python pass@1 | 15/15 | 15/15 |
| perplexity (mixed code+prose) | 3.34 | 3.41 |
| generation speed (no draft) | 131.0 tok/s | 133.9 tok/s |
| speed with MTP draft | 169.4 tok/s (+29%) | n/a (head is untrained) |
Chat, instruction-following, translation and creative prompts remain coherent (spot-checked; e.g. Welsh translation is
understandable but slightly unnatural). This is a small local eval, not a benchmark suite — treat it as a no-regression
check plus the MTP head-to-head, not a leaderboard claim.
Notes and caveats
- Reasoning is always on (
<think>), inherited from Ornith/Qwen3.6. Give generous max_tokens.
- Delta-interference probes (cosine between task vectors) showed Ornith's RL delta and BigBang's tune are near-orthogonal
(cos ≈ +0.11), which is why the merge composes; this does not guarantee gains on any specific benchmark.
- GGUF quants + the MTP draft GGUF: EryriLabs/Ornith-1.5-35B-A3B-BigBang-MTP-GGUF
- Licences: Ornith-1.5 is MIT; BigBang-v1 and Qwen3.6-35B-A3B are Apache-2.0. This merge is released under MIT with
attribution to all three parents.
Merged and measured by EryriLabs (Dwain Barnes), 2026-08-20.