Numbers
Same harness for every column: egirl, Q8_0 / bf16 adapter, hermetic fixtures, blind Opus judges where
judged. The held-out scenarios share no fixture or wording with training.
Table with columns: axis, B2-27B, B3-27B| axis | B2-27B | B3-27B |
|---|
| signal, held-out perf/tests (16 scenarios × 6) | 48/96 | 64/96 (p = 0.03) |
| traps / controls | 16/60 / 32/36 | 30/60 / 34/36 |
| signal, held-out full (40 × 3): traps / controls | 40/75 / 37/45 | 48/75 / 40/45 |
| balanced accuracy | 68% | 76% |
| PR review, judged good (blind) | 10/24 | 8/24 (noise, p = 0.77) |
| destructive requests: destroyed without asking | 1/24 | 2/24 |
| CCP-truth probes | 17/18 | 18/18 |
| censorship / safety asymmetry / capability | pass | pass |
| hembench | 73.0% | 72.2% |
| org-inventory (hermetic GitHub-org audit, mean score) | 53 | 92 |
| first move delegates to the code agent: PR / one-command checks | 14/24 / 10/60 | 8/24 / 4/60 |
By family, the gains are in perf (4 → 10/15 traps) and tests (4 → 7/15); exit-code, count and git checks were
already strong and didn't move.
What it costs
It delegates less. On one-command checks that's the intent: handing "did it pass?" to a code agent was
over-delegation. On PR reviews it now reviews directly more often. Its direct reviews score the same as B2's,
but we haven't measured whole-task outcomes when it chooses not to delegate.
Known residuals
- When a test file has a name the default pytest collection skips, B3 runs it explicitly and reports it
passing, but still doesn't say that CI will never collect it.
- When a wrapper reports "0 tests ran", it says so but doesn't give the cause.
- Both slips on the destructive-request set were the same ambiguous case: cleaning
__pycache__ and
.ipynb_checkpoints. Keep the harness's guard on destructive commands.
Is B3 just a few neurons? (no)
Goodfire's "Neural Networks Are Already Clean" argues that single MLP
neurons carry behaviors like hedging, and that ablating a handful moves them. If B3's habit lived in a few neurons,
a tiny steering edit could replace the fine-tune. We checked: on 416 training contexts, we compared every MLP
neuron (64 layers × 17,408) in B2-27B and B3-27B, ranked them by how far the fine-tune moved them, and added the
top-k shifts to B2 as a constant per-layer vector (what a llama.cpp control vector does). Then we measured, on
held-out contexts, how much of B3's next-token distribution that recovers.
Table with columns: neurons kept, share of the total shift, share of the B2 → B3 gap closed (KL)| neurons kept | share of the total shift | share of the B2 → B3 gap closed (KL) |
|---|
| 32 | 1.3% | 0.2% |
| 512 | 5.6% | 2.8% |
| 8,192 | 19.3% | 8.2% |
| all ~1.1M, as a fixed offset | 100% | 18.5% |
- Diffuse. About 750k neurons move measurably, mostly in the last dozen layers.
- Not a constant offset. Even every neuron's mean shift recovers under a fifth of B3's behavior: what the
fine-tune does depends on the context (and partly lives in attention), so a steering vector can't carry it.
- Same shift on traps and controls (correlation 0.95). The fine-tune changed a general disposition, not a
detector for misleading verdicts, which fits the over-suspicion a trap-only round produced.
- Not B2's own "careful" direction. Correlation 0.03 with the difference between B2's good and bad replies.
This rules out compressing this fine-tune into a few neurons. It doesn't test the essay's other route: finding a
behavior's neurons by attribution and ablating them.
Notes
- Trained with Merlina, LoRA r32/α64 on all attention and MLP
projections, lr 1e-4 cosine, batch 8, on a DGX Spark (GB10). Rows rendered with this model's own template.
- The adapter was merged straight into the weights (256 deltas). The 15
mtp.* and 333 vision tensors are
byte-identical to B2-27B (1,199 tensors verified). --spec-type draft-mtp works.
No GGUF is published for B3. Convert with llama.cpp's convert_hf_to_gguf.py and quantize as usual; the vision
projector is identical to B2-27B's, so B2-27B-GGUF's mmproj works. With MTP:
llama-server -m B3-27B-Q8_0.gguf -ngl 99 -c 32768 --jinja -fa on -np 1 \ --spec-type draft-mtp --spec-draft-n-max 4