Merge Details
Merge Method
- Merge Jackrong/Qwopus3.5-4B-v3 and Jackrong/Qwopus3.5-4B-Coder using the SLERP method → Create Model 1
- Merge BAAI/AREX-Turbo and InternScience/Agents-A1-4B using the SLERP method → Create Model 2
- Merge Model 1 and Model 2 using DARE-TIES → This model
Configuration
I performed the merge according to these YAML files:]
Model 1 (Qwopus3.5-4B-Fusion)
slices:
- sources:
- model: Jackrong/Qwopus3.5-4B-v3
layer_range: [0, 32]
- model: Jackrong/Qwopus3.5-4B-Coder
layer_range: [0, 32]
merge_method: slerp
base_model: Jackrong/Qwopus3.5-4B-v3
parameters:
t:
- filter: self_attn
value: [0.2, 0.3, 0.4, 0.5, 0.6, 0.6, 0.5]
- filter: mlp
value: [0.5, 0.5, 0.5, 0.5, 0.5]
- value: 0.5
dtype: bfloat16
Model 2 (AREX-Turbo + Agents-A1-4B)
slices:
- sources:
- model: BAAI/AREX-Turbo
layer_range: [0, 32]
weight: 0.5
- model: InternScience/Agents-A1-4B
layer_range: [0, 32]
weight: 0.5
merge_method: slerp
base_model: BAAI/AREX-Turbo
parameters:
t:
- value: 0.5
dtype: bfloat16
Mephisto-4B-v2
merge_method: dare_ties
base_model: /path/to/Qwen/Qwen3.5-4B
models:
- model: /home/CloudGoat/llms_merge/Qwents-4B
parameters:
weight: 1.0
density: 1.0
- model: /home/CloudGoat/llms_merge/Qwopus3.5-4B-Fusion
parameters:
weight: 0.7
density: 0.3
parameters:
int8_mask: true
normalize: false
dtype: bfloat16
Benchmarks
In progress