Merge recipe
Stage 1 — DELLA onto Qwen3.6-27B (λ₁ = 1.0)
Two deltas are pruned, sign-elected, and added to the base:
Table with columns: Source, Type, weight, density, epsilon| Source | Type | weight | density | epsilon |
|---|
Ravionhf/qwen3.6-27b-reasoning-distill-lora-v1 | LoRA (r=32, α=64, rsLoRA off → scaling 2.0) | 0.5 | 0.5 | 0.1 |
nerkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill | full weights | 0.5 | 0.6 | 0.1 |
magprune — within each block, entries are ranked by magnitude and assigned a
keep probability p interpolated linearly over [density − ε, density + ε].
A Bernoulli mask is drawn and survivors are rescaled by 1/p, so the pruned
delta stays an unbiased estimator of the original.
Sign election — the two pruned deltas are stacked; the majority sign is
taken per-element and any contribution disagreeing with it is dropped before
summing. This is what keeps two independently-trained reasoning distills from
cancelling each other out.
The LoRA covers all 64 layers (304 modules across in_proj_{a,b,z,qkv},
out_proj, q/k/v/o_proj). It does not touch the MTP head.
Stage 2 — cross-generation task vector (λ₂ = 0.15)
τ = W(nightmedia/Qwen3.5-27B-Engineer-Deckard-Gemini) − W(Qwen/Qwen3.5-27B)
W_final = W_stage1 + 0.15 · τ
This moves an engineering/coding task vector across a model generation. That is
only legitimate if the two generations still share a loss basin — see the
measurement below.
Why these coefficients
Median over 17 automatically-selected probe tensors (MLP down/gate/up,
linear_attn.in_proj_qkv at layers 0/20/42/63, plus embed_tokens), sampled
8 rows each over HTTP Range requests — about 4 MB of traffic total.
Table with columns: #, Pair, cos, ‖Δ‖/‖W‖| # | Pair | cos | ‖Δ‖/‖W‖ |
|---|
| ① | Deckard vs Qwen3.5-27B | 0.9996 | 0.0276 |
| ② | Qwen3.5-27B vs Qwen3.6-27B | 0.9478 | 0.3208 |
| ③ | vision tower: Deckard vs Qwen3.5 | 1.0000 | 0.0018 |
| ④ | nerkyor vs Qwen3.6-27B | 0.9997 | 0.0266 |
Row ② is the gate for Stage 2. cos = 0.9478 between the 3.5 and 3.6 base
weights means the generational update was a large but directionally coherent
move — neuron correspondence survived, so a task vector computed in 3.5 space
still points somewhere meaningful in 3.6 space. If this had come back near
zero cosine, Stage 2 would have been noise injection and was to be disabled.
‖Δ‖/‖W‖ = 0.3208 also sets the ceiling. λ₂ was held at 0.15 rather than
0.3 specifically because the generational drift is already a third of the
weight norm; a larger transplant risks pushing weights outside the basin.
Resulting contribution budget (relative to base weight norm):
Table with columns: Contribution, Product, Share| Contribution | Product | Share |
|---|
| nerkyor | 0.5 × 0.0266 | 0.0133 |
| Deckard τ | 0.15 × 0.0276 | 0.0041 |
| LoRA | 0.5 × 0.0057 | 0.0029 |
Stage 2 sits at 0.31 × the DELLA main ingredient — a supporting voice, not a
second opinion loud enough to overwrite the merge. All three terms are within
one order of magnitude of each other, which is the condition for sign election
to be meaningful rather than a rubber stamp for the largest delta.
What was deliberately left alone
Table with columns: Group, Tensors, Treatment| Group | Tensors | Treatment |
|---|
Vision tower (model.visual.*) | 333 / 0.86 GB | Copied verbatim from base. Rows ③ and ⑤ measured every donor as bit-identical to base here, so there was nothing to merge. |
MTP head (mtp.*) | 1 layer | Excluded from Stage 2. The LoRA never targeted it. |
norm, embed_tokens, lm_head, a_log, dt_bias, conv1d |
Verification
- Per-shard norm gate — every shard checked for median
‖out‖/‖base‖ ∈ [0.95, 1.08]. All 26 passed.
- NaN/Inf assertion on every tensor before serialization. 0 tensors skipped.
- Tensor-by-tensor reconciliation against the base index after upload: 1199/1199 present, 0 extra, 0 shape mismatch, 0 dtype drift, dtype set =
{BF16}.
- Load test —
AutoConfig + AutoTokenizer resolve; 248,077-token vocab; chat template present.
- vLLM generation smoke test on A100-80GB, 13 prompts.
Smoke test observations
Sampling: temperature=0.6, top_p=0.95, top_k=20, max_tokens=8192.
Thinking budget is well-calibrated — the most common failure mode for merged
reasoning models is <think> spam on trivial input. It does not happen here:
Table with columns: Prompt, think tokens| Prompt | think tokens |
|---|
你好 | 27 |
法国的首都是哪座城市? | 28 |
1+1等于几? | 43 |
2 的 3 次方是多少?只要答案,不要解释。 | 50 → answered 8, nothing else |
| CRT congruence problem | 356 |
| 5-person ranking puzzle | 573 |
|
Verifiable answers were correct — CRT problem → 23; ranking puzzle →
B, D, A, E, C; the trigonometric identity was attacked via the triple-angle
form sin3θ = 4 sinθ sin(60°−θ) sin(60°+θ), which is the intended route.
Instruction following held on the "answer only" prompt.
No embedding damage — three Chinese-only prompts (expository writing,
《吕氏春秋》 idiom exegesis, 陶渊明 classical-Chinese gloss) produced fluent output
with no code-switching, no garbled long-tail tokens, no repetition. This was the
specific risk being probed, since embed_tokens and lm_head participate in
the merge.
Code generation intact — LRUCache from scratch (hash map + doubly-linked
list, __slots__, O(1) both ops) and a GIL/race-condition diagnosis with a
threading.Lock fix.
Usage
vLLM
from vllm import LLM, SamplingParams
from transformers import AutoTokenizer
M = "YFC-112358/Qwen3.6-27B-Della-Deckard-v1"
tok = AutoTokenizer.from_pretrained(M)
llm = LLM(model=M, max_model_len=16384, gpu_memory_utilization=0.90)
sp = SamplingParams(temperature=0.6, top_p=0.95, top_k=20, max_tokens=8192)
prompt = tok.apply_chat_template(
[{"role": "user", "content": "你好"}],
tokenize=False, add_generation_prompt=True,
)
print(llm.generate([prompt], sp)[0].outputs[0].text)
from transformers import AutoModelForCausalLM, AutoTokenizer
M = "YFC-112358/Qwen3.6-27B-Della-Deckard-v1"
tok = AutoTokenizer.from_pretrained(M)
model = AutoModelForCausalLM.from_pretrained(M, dtype="bfloat16", device_map="auto")
Notes
- Inherits the base chat template, including the
<think> block. Split on </think> to separate reasoning from the answer.
- BF16 weights, 51.7 GB. Single 80 GB card runs it unquantized; ~24 GB at 4-bit.
- If vLLM fails with
Could not find nvcc, set VLLM_USE_FLASHINFER_SAMPLER=0 before importing vllm.
- The vision tower is untouched base weights, so image behavior should match Qwen3.6-27B exactly.
Limitations
- No quantitative benchmarks have been run. Everything above is structural verification plus a 13-prompt smoke test. Treat capability claims as unproven — the merge is validated as not broken, not validated as better.
- Stage 2 is a cross-generation transplant. It is justified by row ② above, but it remains the least conventional part of the recipe and the first thing to ablate (
λ₂ = 0) if behavior seems off.
- One tensor is absent from the Deckard donor (1198/1199); it falls through to base for that tensor.
- Long-context behavior past 16K was not exercised.
- Vision/multimodal capability was not smoke-tested at all.
Reproduction
Built on a free-tier Colab CPU runtime (12.7 GB RAM, 87 GB disk, no GPU) by
streaming tensors row-wise over HTTP Range requests, so peak memory stays flat
regardless of tensor size and no donor is ever fully resident on disk.
Total download ≈ 230 GB. The merge notebook is included in this repo.
Credits
DELLA: DARE the Extreme — Revisiting Delta-Parameter Pruning (Deep et al., 2024).
Task arithmetic: Editing Models with Task Arithmetic (Ilharco et al., 2023).
中文说明
这是什么
在 Qwen/Qwen3.6-27B 上做的两阶段权重合并：同代的两个推理蒸馏源做 DELLA 融合，
再叠加一个来自 Qwen3.5 微调模型的小幅跨代任务向量，用于移植工程/代码能力。
所有配比系数都来自实测的权重空间距离，不是拍脑袋定的，测量结果全部公开在上面的表里。
配方
Stage 1 — DELLA（λ₁ = 1.0）
Table with columns: 来源, 类型, weight, density, epsilon| 来源 | 类型 | weight | density | epsilon |
|---|
Ravionhf/qwen3.6-27b-reasoning-distill-lora-v1 | LoRA r=32 α=64，scaling 2.0 | 0.5 | 0.5 | 0.1 |
nerkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill | 全量权重 | 0.5 | 0.6 | 0.1 |
magprune 按幅度排名给保留概率，再除以概率做无偏还原；两路 delta 经符号选举后相加，
与多数派符号相反的贡献直接丢弃——这是防止两个独立训练的推理蒸馏互相抵消的关键。
Stage 2 — 跨代任务向量（λ₂ = 0.15）
τ = W(Deckard-3.5) − W(Qwen3.5-27B)
W_final = W_stage1 + 0.15 · τ
为什么是这些系数
关键是②：官方 3.5 与官方 3.6 之间 cos = 0.9478、‖Δ‖/‖W‖ = 0.3208。
余弦仍然很高，说明代际更新虽然幅度大，但方向是连贯的，神经元对应关系没有被打乱——
这才使得「在 3.5 空间算出来的任务向量搬到 3.6 空间仍然指向有意义的方向」成立。
如果这一行的余弦接近 0，Stage 2 就等于注入噪声，应当直接关掉。
同时 0.3208 也定了上限：λ₂ 保守取 0.15 而不是 0.3，因为代际漂移本身已经占到权重范数的三分之一。
最终三项贡献量级：nerkyor 0.0133 / Deckard 0.0041 / LoRA 0.0029，
Stage 2 是 DELLA 主料的 0.31 倍——配角而非主角，且三项在同一数量级内，
符号选举才有意义（否则就退化成给最大的那个 delta 盖章）。
刻意不动的部分
- 视觉塔（333 个张量 / 0.86 GB）直接抄基座：③⑤ 实测三家供体与基座逐字节相同，本来就没有东西可合。
- MTP 头不参与 Stage 2，LoRA 也从未覆盖它。
norm / embed_tokens / lm_head / a_log / dt_bias / conv1d 参与合并但不剪枝：
对归一化系数和门控参数做幅度剪枝是破坏性的。
验证
26 个分片全部通过 ‖out‖/‖base‖ ∈ [0.95, 1.08]；逐张量对账 1199/1199，形状与 dtype 零偏差，
全 BF16；config 与 tokenizer 可加载，chat 模板存在；A100 上 13 条 prompt 的 vLLM 冒烟测试。
冒烟测试重点结论：trivial 问题没有被 <think> 刷屏（「你好」27 token、「法国首都」28 token），
可验证题全对（同余 → 23，排名 → B D A E C，三角恒等式走的是三倍角正路），
三条纯中文长文本流畅无乱码无中英夹杂（这是本次合并动了 embed_tokens 后最该查的风险点），
代码题 LRUCache 与 GIL 竞态诊断均正常。
局限
- 没有跑任何定量基准。 上面全部是结构校验加 13 条冒烟。当前状态是「验证了没坏」，不是「验证了更好」。
- Stage 2 的跨代移植是整个配方里最非常规的一环，行为异常时第一个该消融的就是它（
λ₂ = 0）。
- Deckard 供体缺 1 个张量（1198/1199），该张量回落到基座。
- 16K 以上长上下文、以及多模态能力，均未测试。