Numbers
Propaganda_Eval (803 questions)
Graded with Nous's judge prompt, but by Claude rather than Grok, so the absolute numbers are not comparable with
Nous's published ones. Graders were blind: each saw every model's answer to a question side by side under
shuffled labels, and the key was kept apart. Fact score is 0–5 from the counts of required facts stated, missed or
opposed. "Propaganda" counts answers where the grader quoted official-frame wording the answer adopted.
Table with columns: B4-27B, B5-27B | B4-27B | B5-27B |
|---|
| fact score (0–5) | 3.85 | 4.12 |
| answers adopting the official frame | 2.9% | 1.2% |
| required facts contradicted | 12 | 4 |
| answers scoring 0 | 73 | 41 |
Paired difference with B4: +0.26 (95% bootstrap CI +0.17 to +0.35). B5 is ahead in every category; the
largest gain is state ideology (3.47 → 3.95), then freedom of expression (3.18 → 3.54) and data privacy and
surveillance (4.10 → 4.43).
For reference, an earlier blind round on the same 803 questions with the same graders' rules: stock Qwen3.8-27B
3.63 (9.8% frame), B3-27B 3.90, B4-27B 3.92, Arete-27B 4.20 (0.25%). So B5 gets most of the way from B4 to Arete;
B4 itself, after several merges and fine-tunes, already sat well above stock Qwen.
B4's reason to exist. Thinking off unless noted, through egirl's
OpenAI-compatible API with web_search, web_research and calculate. Both models were run as Q8_0 GGUFs in the
same session.
Table with columns: right first move, B4-27B, B5-27B| right first move | B4-27B | B5-27B |
|---|
| held-out (41) | 36 | 35 |
| held-out (41), thinking low | 37 | 40 |
| harder set (37) | 32 | 34 |
| keyed answers right (held-out) | 20/20 | 20/20 |
These are single runs; a gap of one or two items is noise. B4 scores lower here than on its own card (41/41) because that card
measured B3 + the f16 adapter, and this is the merged Q8_0 with a newer egirl. Read the columns against each other.
General and code
Table with columns: B4-27B, B5-27B | B4-27B | B5-27B |
|---|
| ARC-Challenge / ARC-Easy / TruthfulQA (MC, llama-perplexity) | 63.5 / 85.6 / 39.4 | 66.2 / 87.2 / 41.0 |
| HemBench (38 tasks, two runs) | 75.7 / 72.8 | 69.2 / 73.0 |
| mean answer length on Propaganda_Eval | 688 chars | 751 chars |
HemBench moves about three points between identical runs at temperature 0.2. B5 is within that.
How we got here
Arete and B4 are the same architecture, so the merge was a short series of experiments. Each candidate was screened
on a 200-question stratified subset with blind side-by-side grading (B4 and Arete included as anchors in every
round), and the survivors ran the full 803, the egirl tool-use sets and the general gates.
Table with columns: recipe, fact (200), frame| recipe | fact (200) | frame | |
|---|
| B4-27B | 3.82 | 3.5–5% | anchor |
| SLERP B4→Arete t=0.25 | 3.98 | 2.0% | |
| SLERP B4→Arete t=0.5 | 4.11 | 1.0% | full run 4.06; answers 70% longer, no better than task arithmetic |
| TIES (density 0.5) | — | — | broken: outputs an empty </think> |
| task arithmetic, Arete × 0.5 | 4.10 / 4.16 | 2.5% / 3.5% | B5 |
| task arithmetic, Arete × 1.0 | 4.04 | 0% | |
| task arithmetic, Arete × 1.5 | 4.12 | 0.5% | full run 4.12, 0.4% frame; but 7 empty answers, weaker thinking-off tool use, −2 HemBench tasks |
| task arithmetic, Arete × 2.0 | 3.70 | 0% | 23/200 answers think and then stop |
| Arete-27B | 4.24 / 4.29 | 0–0.5% | anchor |
What we learned:
- Arete is a small move; B4 is a big one. Arete's SFT moved Qwen's weights by only 0.3–0.6% (relative norm
per tensor). B4, after its merge-and-fine-tune lineage, sits 20–37% away from Qwen. So SLERP from B4 "toward
Arete" is really SLERP back toward stock Qwen, and TIES with density pruning wrecks B4's large delta. Task
arithmetic adds Arete's delta on top of B4's and leaves B4 intact.
- The fact gain saturates early; the frame keeps falling. Facts plateau around half of Arete's delta, while
adopting the official frame keeps dropping with dose. Past 1× the model starts ending its thinking without an
answer. We took the smallest dose with the full fact gain and no breakage.
- The merge keeps B4's thinking length. All merges think for roughly 430–650 tokens per Propaganda_Eval answer,
like B4; Arete thinks for about 3,200.
Contamination check
We checked the training data we hold for the B-series 27B line against Propaganda_Eval using 8-gram shingles of each
prompt and its required facts: 0 of 803 items overlap. The data behind the upstream merge donors in B4's lineage is
not ours to inspect.
Notes
- Arete lacks Qwen's 15 multi-token-prediction (
mtp.*) tensors. Kessel merges a tensor with whichever donors have
it, so B5's MTP head is B4's.
- Tokenizer and chat template are B4's (Qwen3.8's). Vision weights are merged like everything else; vision was
not evaluated.
- Built for egirl. Other harnesses weren't a goal and weren't tested.
- B4's known residual carries over: it sometimes writes math in LaTeX.
Merge config
models:- model: schneewolflabs/B4-27B parameters: weight: 1.0- model: NousResearch/Arete-27B parameters: weight: 0.5merge_method: task_arithmeticbase_model: Qwen/Qwen3.8-27Bdtype: bfloat16