Merge configuration
Table | |
|---|
| Method | TIES (magnitude sparsification + sign-consensus, delta normalization) |
| Density | ρ = 0.3 |
| Weights | uniform (λ = 1 for each of the three task vectors) |
| Base / anchor | Qwen/Qwen2.5-14B-Instruct |
| Precision | BF16 throughout |
| Result | 579 tensors, 14.77B parameters |
Task vectors are computed relative to the pretrained base, τᵢ = φᵢ − φ₀, and the merged
model is φ_merged = φ₀ + TIES({τᵢ}).
Sources
Merged with lfmerge:
python main.py --method ties \
--models <nqa> <msq> <bcp> \
--weight 1 1 1 --density 0.3 0.3 0.3 \
--base Qwen/Qwen2.5-14B-Instruct \
--anchor-base
Merge diagnostics
The three task vectors are close to orthogonal, which is the low-interference regime TIES
is designed for. Measured on representative weight matrices:
Table with columns: Quantity, Value| Quantity | Value |
|---|
| pairwise cos(τᵢ, τⱼ) | +0.002 to +0.038 |
| all-three sign agreement | 0.24 (chance level for independent signs is 0.25) |
| ‖τ‖/‖φ₀‖ | 5.5–5.8% (NQA), 6.4–6.6% (MSQ), 7.6–7.7% (BCP) |
After merging, all three task vectors are retained with positive alignment:
Table with columns: cos(φ_merged − φ₀, τᵢ) | cos(φ_merged − φ₀, τᵢ) |
|---|
| NarrativeQA | +0.28 to +0.37 |
| MuSiQue | +0.43 to +0.46 |
| BrowseComp-Plus | +0.57 to +0.62 |
BrowseComp-Plus is retained most strongly, consistent with it having the largest task-vector
norm (it was trained on the most QA pairs). Weight integrity was checked across all 579
tensors: no non-finite values, and all shapes and keys match the base model.
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "yungisimon/Qwen2.5-14B-ties-merge-nqa-msq-bcp"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")
Or with vLLM:
vllm serve yungisimon/Qwen2.5-14B-ties-merge-nqa-msq-bcp --dtype bfloat16 --max-model-len 32768
Status and limitations
- Not yet evaluated. Downstream accuracy on the three corpora has not been measured for
this merge; the diagnostics above are parameter-space measurements only.
- Uniform merge weights were used, so the corpus with the largest task-vector norm
(BrowseComp-Plus) contributes most. Weighting by inverse task-vector norm is untested.
- ρ = 0.3 was chosen because it was the best-performing density in prior merge sweeps on
these checkpoints; it was not re-tuned for this three-corpus setting.
- The three corpora are treated as pairwise disjoint.
License
Inherits the license of Qwen/Qwen2.5-14B-Instruct (Apache 2.0).