Measured
No capability damage. ARC-Challenge (299 tasks, paired McNemar against the base) is flat at
every scale, despite this being the largest adapter in the family by effective delta:
Table with columns: scale, ARC, delta, p| scale | ARC | delta | p |
|---|
| 0.25 | 64.88 | +0.00 | 1.000 |
| 0.50 | 64.55 | −0.33 | 1.000 |
| 0.75 | 65.22 | +0.33 | 1.000 |
| 1.00 | 64.55 | −0.33 | 1.000 |
Worth noting against the family: ‖B·A‖ = 18.95 here, versus 7.81 for a sibling adapter that
cost 4 ARC points at scale 1.0 (p = 0.036). Delta magnitude does not predict damage — what the
adapter changes matters more than how much. This one moves tool-call behaviour, which ARC never
exercises.
Agent behaviour changed, mostly for the worse. Ten held-out tasks from a repository that
contributed no training data, graded by running the repo's own test suite:
Table with columns: outcome, Wichtel, +Lehrling| outcome | Wichtel | +Lehrling |
|---|
| passed | 9/10 | 7/10 |
| self, passed | 7 | 7 |
| escalated, passed | 2 | 0 |
| self, failed | 0 | 3 |
| escalated, failed | 1 | 0 |
Every task the base delegated, this attempts alone. Two of those turned out fine — including one
the base had escalated and still failed. Three did not, and they failed in the same way.
Thinking mode is untouched. An 8-prompt probe at temperature 0, matched 12k token cap:
Table with columns: Wichtel, +Lehrling | Wichtel | +Lehrling |
|---|
| unclosed think blocks | 0/8 | 0/8 |
| total reasoning | 6,872 words | 6,914 (1.01x) |
This was not the expected result. The training targets contain no <think> blocks at all —
Claude Code does not export thinking — so 372 steps of never deliberating looked likely to
suppress it. It did not, and the probable reason is format_type: raw: the prompts are
pre-rendered conversations that never contain the <think> opener, so the model was never shown a
context where it sat inside a think block and produced nothing. The reasoning pathway was simply
never in the gradient.
Trained through a chat template with enable_thinking: true instead, the same data would likely
suppress reasoning — the model would see the opener followed by an immediate close, 372 times. The
raw format isolated the two behaviours by accident.
Practically: this composes with a reasoning adapter rather than competing with one.
The failure mode
The three regressions are not scattered. All three explored and never edited:
metrics_words: 10 tools — read, glob, execute x5 ... no write, no edit
"[Agent reached maximum turns without producing a final response]"
metrics_content_words: 6 tools — read, glob, read, read, git_log, read
(empty response)
The function under test was left holding raise NotImplementedError, untouched.
This is a data composition problem rather than a training failure. Claude Code explores heavily
before committing to an edit, and with one sample per assistant turn that exploration dominates
the dataset — Read, Glob, Grep, Bash vastly outnumber Write, Edit and Agent. The
adapter learned the reconnaissance half of the behaviour and not the decisive half.
The obvious next experiment: filter the source dataset to samples whose target is a mutating
action — Write, Edit, or Agent — and retrain. That is a filter on the existing 24k, not new
data, and it is testable on the same ten tasks.
Honesty about the statistics
Three lost, two won on ten tasks is p = 1.000. Nothing here is statistically significant, and
the direction should not be over-read. What raises it above noise is that the three failures share
one mechanism and one file family rather than scattering — a pattern that usually survives more
data, though that remains to be shown.
Overfitting
Not strongly indicated, with one caveat: no eval loss was recorded despite eval_steps: 0.5,
so the standard check was unavailable.
- No format bleed. Across ten agent runs, zero responses contained Claude Code tool syntax and all
ten used the host harness's native tool names.
- Verbatim memorisation probe on twelve training prompts at temperature 0: mean similarity 0.496,
with 2/12 above 0.9. Both high scores are tool-call structures, where the syntax is rigid and
much of the similarity is template rather than recall.
- Training loss fell smoothly, 1.508 → 0.987, no divergence.
Training
Table | |
|---|
| base | nbeerbower/Wichtel-Qwen3.6-27B |
| method | SFT, LoRA r=32 α=64 dropout 0.05 |
| data | 6,000 shuffled samples of 24,395 |
| schedule | 1 epoch, 372 steps, lr 2e-4 cosine, warmup 0.05 |
| batch | 1 × grad-accum 16 |
| sequence | max_length 4096, max_prompt_length 3600 |
| format | raw — the prompts are pre-rendered Qwen3 conversations |
The learning rate was too high. Loss bottomed at 0.947 by step 16, climbed to 1.212 by step 56
as the LR sat near its 2e-4 peak, and only recovered as cosine decay took hold — roughly the first
120 steps were spent undoing the first 40. 5e-5 would likely reach a lower final loss in the same
budget. The 2e-4 was inherited from a sibling run that was installing a capability the base
lacked entirely, where aggressive updates are appropriate; nudging behaviour a model already has
is a different problem.
Subsampled to 6,000 because the full 24,395 samples at ~3,300 tokens each is ~79M tokens per
epoch, which on the training hardware is 15 days. At 6,000 it is 3.3 days.