The problem it fixes
Stock Gemma-4-26B-A4B-it is a strong agent until a task gets long. Then it can fall into a loop — the same paragraph
of reasoning, or the same tool call, again and again until the context or the wall clock runs out. This model was
trained to keep working instead.
Measured head-to-head against the base model on a long agentic repair task, 20 runs each with identical settings:
12 of 20 runs finished within the 30-minute budget against 3 of 20 for the base model, with
reasoning loops cut about 6× less often. Details below.
Head-to-head with the base model
Table with columns: base gemma-4-26B-A4B-it, this model | base gemma-4-26B-A4B-it | this model |
|---|
| runs finished within the 30-minute limit (20 each) | 3 / 20 (15 %) | 12 / 20 (60 %) |
| 95 % interval (Wilson) | 5 %–36 % | 39 %–78 % |
| passed within 10 / 20 / 30 min | 0 % / 15 % / 15 % | 25 % / 50 % / 60 % |
| reasoning-loop cuts per run (median) | 8.0 | 1.0 |
| reasoning-loop cuts per model-hour | 18.1 | 2.9 |
Pass/fail difference: Fisher exact two-sided p = 0.0079.
What the pass rate means. It is not an absolute success rate: it is the share of runs that finished the repair
within a fixed 30-minute budget. The test is built to measure what the fine-tune changes in a definite amount of
time — how much of the budget goes to loops, how quickly the model gets from a broken file to a working game. A timeout
means "not done in 30 minutes", not "cannot do it". With more time both models would pass more often and the gap would
likely narrow, but not by much: in extended manual testing of the base model and of its pruned 98-expert variants
(the v7-coder and v9-agentic recipes), a run that does not converge quickly becomes steadily more likely to loop, so
the base model either needs one to two hours or more to pass, or ends in massive loops. Anti-repetition sampling hides
the loops but costs the model much of its effectiveness and its ability to converge. The fine-tune targets exactly
this — less looping, faster convergence — and the data below shows it achieved. With only presence_penalty 0.025
(no other anti-repetition setting) it is virtually loop-free and completes the task in an acceptable time even on
runs where it does not converge quickly. The
pass curve below shows how the rate builds up minute by minute.
Setup. One task, run 20 times per model: repair a broken HTML game (manic_miner, a syntax error plus logic
bugs). The runs are orchestrated and scored by automaticum, a test harness
for coding agents whose oracle runs the game itself. The agent is the
Cerebriline SDK 4.100.277 (VS Code host profile, LSP diagnostics after every edit
and the check_file / ask_lsp tools on, browser off) until the game's own oracle reports that it loads, reaches play
and passes 23 logic checks. 30 minutes wall clock per run, one run at a time per model. Both models as BF16 GGUF on the
same llama.cpp build, each on its own RTX PRO 6000 Blackwell (96 GB), f16 KV cache, Gemma-4 assistant drafter for
speculative decoding, thinking on (effort high), temperature 0.8 / top_p 0.95 / top_k 64 / min_p 0 /
presence_penalty 0.025. The two configurations differ only in the model file: harness, task, sampler, server build,
chat template and guards are identical (checked from each run's recorded provenance). The harness guards were on for
both: a reasoning block that repeats itself is cut and the request retried, a repeated identical tool call draws
warnings, then a stop. Arms, server launchers, analysis scripts, the full report and per-run data:
omk/models/gemma-4-26b-a4b/sft11/eval/.
Outcome
Table with columns: base, this model | base | this model |
|---|
| passed | 3 | 12 |
| failed: game never starts (syntax / runtime error left in the file) | 17 | 6 |
| failed: game starts, some logic checks still failing | 0 | 2 (17/23, 21/23 checks) |
| runs ended by the harness for repeating a tool call | 1 | 1 |
| runs that claimed success but failed | 0 | 0 |
| time to the first passing game check, median (passed runs) | 16.9 min | 11.5 min |
Share of all runs passed by minute t:
Table with columns: 5, 10, 15, 20, 25, 30 | 5 | 10 | 15 | 20 | 25 | 30 |
|---|
| base | 0 % | 0 % | 0 % | 15 % | 15 % | 15 % |
| this model | 5 % | 25 % | 40 % | 50 % | 55 % | 60 % |
Loops and guards
Medians per run, all 20 + 20 runs; p = Mann–Whitney two-sided.
Table with columns: base, this model, p | base | this model | p |
|---|
| reasoning blocks cut for repeating themselves | 8.0 | 1.0 | < 0.001 |
| guard interventions, all kinds | 9.5 | 4.0 | 0.001 |
| model requests cut or failed | 9.0 | 1.0 | < 0.001 |
| reasoning blocks with a phrase repeated ≥ 10× | 2.0 | 1.0 | 0.03 |
| worst repeat count in any reasoning block | 12 | 10 | 0.05 |
| repeated-tool-call warnings | 0.5 | 1.5 | 0.12 |
| runs with no reasoning-loop cut | 0 / 20 | 9 / 20 | |
Per model-hour (sum over all runs ÷ total model time): reasoning-loop cuts 18.1 → 2.9,
all guard interventions 24.8 → 17.7, repeated-tool-call warnings
6.1 → 9.9.
Time, length and effort
Table with columns: base, this model, p | base | this model | p |
|---|
| wall time per run (min, 30 = timeout) | 30.2 | 21 | 0.05 |
| longest single model turn (s) | 213 | 108 | 0.001 |
| median model turn (s) | 22 | 9.1 | 0.006 |
| reasoning per run (k characters) | 226 | 230 | 0.95 |
| longest reasoning block (k characters) | 38 | 30 | 0.005 |
| generated tokens per run (k) | 121 | 128 | 0.56 |
| tokens written into tool-call arguments (k) | 12.1 | 22.6 | 0.17 |
| generated tokens per tool call | 3,866 | 2,100 | < 0.001 |
| peak context (k tokens) | 84 | 110 | 0.26 |
Same restricted to failed runs (all or nearly all of the 30 minutes used): reasoning 232k vs
416k characters, tool calls 24 vs 69,
reasoning-loop cuts 9.0 vs 1.0.
Behaviour
Table with columns: base, this model, p | base | this model | p |
|---|
| tool calls per run | 28 | 50 | < 0.001 |
| tool calls per model-minute | 0.9 | 3.3 | < 0.001 |
| edits applied | 5 | 10 | < 0.001 |
| edits refused by the tool | 3 | 7 | 0.004 |
| edits not followed by a check | 4 | 6 | 0.08 |
game runs (run_game) | 2 | 5 | < 0.001 |
read_files calls | 8 | 13 | 0.006 |
| LSP: edits checked automatically | 8 | 18 | < 0.001 |
LSP: check_file calls | 1 | 0 | 0.58 |
LSP: ask_lsp calls | 0 | 0 | 0.35 |
| runs that restored the original file and started over | 7 / 20 | 12 / 20 | |
What changes. Total reasoning per run is about the same (226k vs 230k
characters); how it is spent is not. The base model spends 3,866 generated tokens per tool
call against 2,100, its longest turn runs 213 s against
108 s, and its reasoning repeats itself until the guard cuts it (8.0
cuts per run against 1.0). It makes 0.9 tool calls per
model-minute against 3.3 and runs the game 2 times per
run against 5, so it sees far less often what is still broken. All 17
of its failed runs ended with the game not even loading; 2 of this model's 8 failures got
the game running and passing 17/23, 21/23 logic checks.
Known weaknesses
- Repeating a refused tool call. When an edit is refused, this model is more likely than the base model to send the
same call again: more repeated-tool-call warnings in failed runs (median: base 0.0,
this model 6.0; p = 0.010). Runs ended by the harness for
repeating a call: base 1, this model 1.
- Degenerate tool-call bursts. In 2 of 20 runs this model emitted the same refused
editor call
310 and 320 times in a single response, until the output-token cap. Both began with an
edit whose start_line argument was dropped, so the tool refused it (the base model hit that refusal in 7 runs, this model in
7); the base model never answered it with a burst. Worth capping tool calls per response when running it in an
agent loop.
- Serving. This model's server generated 168 tok/s against 176 for the base model's
(4 % slower on the same build and GPU model), which if anything works against it under a
wall-clock limit. Speculative-draft acceptance was the same for both (90 % vs 89 %), so it is not the drafter; this model also worked at longer contexts (median peak 110k vs 84k tokens), which slows decoding.
- Scope. One task, 20 runs per model, one harness. The intervals above are wide; treat the size of the gap as
approximate, its direction as solid.
Earlier gates
Before the head-to-head, the same task ran 10 times per checkpoint (automaticum, Cerebriline SDK 4.100.266) at temperature 1.0
(top_p 0.95, top_k 64, min_p 0), 30 minutes per run:
Table with columns: base, step 400, step 800, final (this model) | base | step 400 | step 800 | final (this model) |
|---|
| runs finished within 30 min | 1 / 10 | 4 / 10 | 5 / 10 | 5 / 10 |
| loops per model-hour (guard cuts + loop blocks) | 9.2 | 9.7 | 3.1 | 2.0 |
On SDK 4.100.276 with the recommended sampler, 7 of 8 finished runs passed. In a VS Code agent session on the same
server it fixed the game in 3 of 3 sessions in 6–10 minutes, with no loops seen.
How it was trained
Built with omk-model-optimizer in its BF16 recovery mode
(guide): the model's own
bf16 weights are trained directly (no quantizer, no LoRA, no fp32 master copy), and the export writes the source
checkpoint back with only the trained tensors replaced.
Data — mostly on-policy. Prompts, tool declarations and recorded tool outputs come from public SFT sets (9 NVIDIA
Nemotron sets and interstellarninja/hermes_reasoning_tool_use). Every trained assistant turn was regenerated by the
untouched Gemma-4-26B-A4B-it itself, turn by turn, in its own chat template with thinking on, and kept only when a
validator passed it: no reasoning loop, no foreign tokens, a tool call equivalent to the recorded one (so the
recorded tool output still answers it), the right final answer where a reference exists. One source is regenerated by
the bigger sibling Gemma-4-31B-it (Kimi-K2.5 agent tasks from lambda/hermes-agent-reasoning-traces). The model
learns to finish tasks in its own style rather than imitating another model's.
Table with columns: corpus| corpus | |
|---|
| sources | agentic, opencode, hermes, ifchat v1/v2, kimi31b (Gemma-4-31B), swe, multilingual, science, math, competitive programming |
| size | 16,296 training conversations → 4,217 packed windows of 16,384 tokens (11.6 M scored tokens), one epoch |
| held out | 522 conversations (per-source CE every 200 steps) + 27 agentic windows scored against Gemma-4-31B |
| published | ManniX-ITA/gemma-4-26B-A4B-agentic-sft (conversations + the exact training windows) · ManniX-ITA/hermes-agent-kimi-traces-gemma4-31b |
Training.
Table | |
|---|
| trained | attention, per-layer dense MLP, router and all routed experts: 295 tensors, 24.5 B parameters (embedding, norms, vision tower frozen) |
| loss | next-token cross-entropy on the regenerated assistant tokens only (reasoning, prose, tool calls, stop token); prompts, user turns and tool outputs masked; each packed conversation attends only to itself |
| schedule | 2,108 steps × batch 2 × 16,384 tokens, peak LR 4.74e-6 (10-step warmup, cosine), momentum-free Adafactor with stochastic-rounding bf16 updates |
| hardware | one RTX PRO 6000 Blackwell (96 GB), 74 GiB peak, 25.5 h |
| result | held-out CE on the corpus 0.281 → 0.263; agreement with Gemma-4-31B on held-out agentic windows unchanged (top-1 0.930 → 0.929) |
Full recipe, every script as run, corpus provenance (upstream revisions, sha256 of every file) and reproduce commands:
omk/models/gemma-4-26b-a4b/
in omk-model-optimizer. The three training modes (QAD to GGUF, ternary recovery, BF16 recovery) are documented in
omk/docs/.
Use
Exactly like google/gemma-4-26B-A4B-it (same chat_template.jinja, processor and generation config; thinking via
enable_thinking).
Recommended sampler for agentic work (the setting used for the head-to-head): temperature 0.8, top_p 0.95, top_k 64,
min_p 0, presence_penalty 0.025, no repetition penalty.
from transformers import AutoProcessor, AutoModelForMultimodalLMmodel_id = "ManniX-ITA/gemma-4-26B-A4B-agentic"processor = AutoProcessor.from_pretrained(model_id)model = AutoModelForMultimodalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto")
vLLM:
vllm serve ManniX-ITA/gemma-4-26B-A4B-agentic --dtype bfloat16 --enforce-eager --max-num-batched-tokens 8192 --reasoning-parser gemma4 --default-chat-template-kwargs '{"enable_thinking": true}'
— the settings the training
corpus was generated with on vLLM 0.20.2 (
--enforce-eager avoids a Gemma-4 MoE routing fault under CUDA-graph
capture seen on that version). Tool calls were parsed there with a hardened Gemma-4 parser plugin, published in the
recipe as
sft11/corpus/sft2/gemma4_safe_tool_parser.py
(
--enable-auto-tool-choice --tool-call-parser gemma4_safe --tool-parser-plugin gemma4_safe_tool_parser.py).
Limitations
- Evaluated on agentic coding behaviour (the head-to-head above) and held-out CE; no general benchmark suite has been
run on this checkpoint yet. 8 of 20 head-to-head runs still did not finish within 30 minutes: reasoning
loops are much rarer, not gone, and repeated refused tool calls are a known failure mode (see Known weaknesses).
- Trained on English-dominant agentic, coding and instruction-following data, with a small multilingual share; vision
was not trained (the tower is the base model's, byte for byte).
License
Apache-2.0, as the base model. Training data licenses: see the corpus dataset card.