Recipe
python main.py \
--models <nqa> <msq> <bcp> \
--base Qwen/Qwen2.5-14B-Instruct \
--method ties_sum --weight 1 1 1 --density 0.2 0.2 0.2 --raw-weights --anchor-base
Task vectors are computed against the pretrained base, tau_i = phi_i - phi_0.
Table | |
|---|
| alpha = <D_merged, tau_i>/||tau_i||^2 (nqa / msq / bcp) | 0.69 / 0.71 / 0.75 |
| ||D_merged|| / ||phi_0|| | 9.0% |
| single-corpus fine-tune, for reference | 4.8–7.0% |
alpha = 1.0 would mean a corpus's fine-tuning survived the merge at full strength.
Measured result
Single-turn knowledge recall: the MEMORY model is asked corpus questions directly, with no
multi-turn protocol, judged against gold answers by Qwen2.5-32B-Instruct. n=80 per corpus,
restricted to the subsets each model was actually trained on.
Table with columns: model, BrowseComp-Plus, MuSiQue| model | BrowseComp-Plus | MuSiQue |
|---|
| this checkpoint | 8.8% | 13.8% |
yungisimon/Qwen2.5-14B-ties-merge-nqa-msq-bcp (recommended) | 16.2% | 25.0% |
| per-corpus specialist (ceiling) | ~37% | ~36% |
| untrained Qwen2.5-14B-Instruct (floor) | 0% | ~2.5% |
The reference figures are from the same probe run as this checkpoint, because cross-run
noise is about +/-5pp at n=80 (the reference measured 21.2 / 16.2 / 25.0 on BCP across three
runs). Differences smaller than ~5pp are not resolvable at this sample size.
Sparse summing with consensus. Also loses to averaging.
Why these all failed
The three task vectors are near-orthogonal in aggregate (pairwise cosine 0.002–0.038), which
suggests summing them rather than averaging, since orthogonal updates should not interfere.
That reasoning is wrong: 75.4% of coordinates carry a sign conflict between the three
vectors. Global orthogonality does not imply coordinate-wise compatibility. TIES's magnitude
trimming and sign-consensus filtering are therefore doing necessary conflict resolution, not
merely attenuating the task vectors — so recipes that preserve more of each task vector
(higher alpha) consistently do worse. Six candidates across four families (pure sum, sparse
sum, sub-0.3 sparsity, non-uniform weighting) all lost to plain TIES rho=0.3 with uniform
averaging.
Caveats
- The probe asks the full question in a single turn, whereas the MEMORY model is trained
on decomposed, self-contained reflection pairs and is used at inference on decomposed
sub-queries. Absolute numbers are therefore well below published protocol accuracy and are
not comparable to it; the probe is a relative screen of knowledge accessibility.
- Judging is a single greedy pass of one local model, not human-validated.
- A NarrativeQA column was collected but discarded: the NQA specialist scored 0% on its own
corpus, so that probe measures prompt-style mismatch rather than knowledge.
License
Inherits the license of Qwen/Qwen2.5-14B-Instruct (Apache 2.0).