Merge Details
Merge Method
This model was merged using the DELLA merge method using TheDrummer/Precog-24B-v1 as a base.
Models Merged
The following models were included in the merge:
- BeaverAI/Fallen-Mistral-Small-3.1-24B-v1e
- BeaverAI/Cydonia-24B-v4x-GGUF
- mradermacher/BlackXorDolphTronGOAT-GGUF
- TheDrummer/Cydonia-24B-v4.3
- TheDrummer/Precog-24B-v1
Configuration
The following YAML configuration was used to produce this model:
architecture: MistralForCausalLMbase_model: B:\24B\!models--TheDrummer--Precog-24B-v1models: - model: B:\24B\!BeaverAI_Fallen-Mistral-Small-3.1-24B-v1e_textonly parameters: weight: - filter: "lm_head" value: 0.31 - filter: "o_proj" value: 0.5 - filter: "mlp" value: 0.31 - filter: "embed_tokens" value: 0.31 - value: 0.31 density: 1.0 epsilon: 0.09 - model: B:\24B\!models--darkc0de--BlackXorDolphTronGOAT parameters: weight: - filter: "lm_head" value: 0.07 - filter: "o_proj" value: 0.1 - filter: "mlp" value: 0.07 - filter: "embed_tokens" value: 0.07 - value: 0.07 density: 1.0 epsilon: 0.09 - model: B:\24B\!models--TheDrummer--Cydonia-24B-v4.3 parameters: weight: - filter: "lm_head" value: 0.31 - filter: "o_proj" value: 0.5 - filter: "mlp" value: 0.31 - filter: "embed_tokens" value: 0.31 - value: 0.31 density: 1.0 epsilon: 0.09 - model: B:\24B\Cydonia-24B-v4x-Q8_0 parameters: weight: - filter: "lm_head" value: 0.31 - filter: "o_proj" value: 0.5 - filter: "mlp" value: 0.31 - filter: "embed_tokens" value: 0.31 - value: 0.31 density: 1.0 epsilon: 0.09merge_method: dellaparameters: lambda: 1.0 normalize: false int8_mask: false rescale: truedtype: float32out_dtype: bfloat16tokenizer: source: unionchat_template: auto
Audit
[DELLA Audit] Layer: lm_head.weight | Lambda=1.00 [BASE] !models--TheDrummer--Precog-24B-v1 !BeaverAI_Fallen-Mistral-Small-3.1-24B-v1e_textonl: █████████████████ 34.2% (W:0.31 D:1.00 N:4.43 E:0.09) !models--TheDrummer--Cydonia-24B-v4.3 : ███████████ 24.0% (W:0.31 D:1.00 N:3.10 E:0.09) !models--darkc0de--BlackXorDolphTronGOAT : ████████ 17.1% (W:0.07 D:1.00 N:9.78 E:0.09) Cydonia-24B-v4x-Q8_0 : ████████████ 24.8% (W:0.31 D:1.00 N:3.21 E:0.09) [DELLA Audit] Layer: model.layers.17.mlp.up_proj.weight | Lambda=1.00 [BASE] !models--TheDrummer--Precog-24B-v1 !BeaverAI_Fallen-Mistral-Small-3.1-24B-v1e_textonl: █████████████ 27.0% (W:0.31 D:1.00 N:1.87 E:0.09) !models--TheDrummer--Cydonia-24B-v4.3 : ████████████ 25.5% (W:0.31 D:1.00 N:1.76 E:0.09) !models--darkc0de--BlackXorDolphTronGOAT : ███████████ 23.0% (W:0.07 D:1.00 N:7.05 E:0.09) Cydonia-24B-v4x-Q8_0 : ████████████ 24.5% (W:0.31 D:1.00 N:1.70 E:0.09) [DELLA Audit] Layer: model.layers.17.self_attn.k_proj.weight | Lambda=1.00 [BASE] !models--TheDrummer--Precog-24B-v1 !BeaverAI_Fallen-Mistral-Small-3.1-24B-v1e_textonl: ██████████████ 29.1% (W:0.31 D:1.00 N:0.71 E:0.09) !models--TheDrummer--Cydonia-24B-v4.3 : █████████████ 27.6% (W:0.31 D:1.00 N:0.67 E:0.09) !models--darkc0de--BlackXorDolphTronGOAT : ████████ 16.6% (W:0.07 D:1.00 N:1.79 E:0.09) Cydonia-24B-v4x-Q8_0 : █████████████ 26.8% (W:0.31 D:1.00 N:0.65 E:0.09) [DELLA Audit] Layer: model.layers.17.self_attn.o_proj.weight | Lambda=1.00 [BASE] !models--TheDrummer--Precog-24B-v1 !BeaverAI_Fallen-Mistral-Small-3.1-24B-v1e_textonl: █████████████ 27.0% (W:0.50 D:1.00 N:0.93 E:0.09) !models--TheDrummer--Cydonia-24B-v4.3 : ████████████ 25.4% (W:0.50 D:1.00 N:0.87 E:0.09) !models--darkc0de--BlackXorDolphTronGOAT : ███████████ 23.2% (W:0.10 D:1.00 N:3.99 E:0.09) Cydonia-24B-v4x-Q8_0 : ████████████ 24.4% (W:0.50 D:1.00 N:0.84 E:0.09) [DELLA Audit] Layer: model.layers.17.self_attn.q_proj.weight | Lambda=1.00 [BASE] !models--TheDrummer--Precog-24B-v1 !BeaverAI_Fallen-Mistral-Small-3.1-24B-v1e_textonl: ██████████████ 28.1% (W:0.31 D:1.00 N:1.43 E:0.09) !models--TheDrummer--Cydonia-24B-v4.3 : █████████████ 26.8% (W:0.31 D:1.00 N:1.36 E:0.09) !models--darkc0de--BlackXorDolphTronGOAT : █████████ 19.2% (W:0.07 D:1.00 N:4.32 E:0.09) Cydonia-24B-v4x-Q8_0 : ████████████ 25.9% (W:0.31 D:1.00 N:1.32 E:0.09) [DELLA Audit] Layer: model.layers.17.self_attn.v_proj.weight | Lambda=1.00 [BASE] !models--TheDrummer--Precog-24B-v1 !BeaverAI_Fallen-Mistral-Small-3.1-24B-v1e_textonl: █████████████ 26.3% (W:0.31 D:1.00 N:0.49 E:0.09) !models--TheDrummer--Cydonia-24B-v4.3 : ████████████ 24.7% (W:0.31 D:1.00 N:0.46 E:0.09) !models--darkc0de--BlackXorDolphTronGOAT : ████████████ 25.4% (W:0.07 D:1.00 N:2.09 E:0.09) Cydonia-24B-v4x-Q8_0 : ███████████ 23.6% (W:0.31 D:1.00 N:0.44 E:0.09)