Results
Test sets (the number that matters)
14 tasks x 2 runs on obscura, starting at the store's homepage, sites never trained on. Corrected 2026-10-14: every
check is exact now (the final address must end with the target product's path). Two timber products are not sold on
their site (Bunnings, NZ Timber): those tasks pass only when the agent gives up on its own. The first version of this
card used looser checks that counted a search page and a 138x32 deck as found (r1 22/28, prod 10/28).

Table with columns: Kev alone, cascade (Kev, escalates to holo3.1) | Kev alone | cascade (Kev, escalates to holo3.1) |
|---|
Kev prod (jaredpalmer/kev-4b) | 14/28 | 16/28 |
| r0b + guard-v2 | 16/28 | 15/28 |
| r1 + guard-v2 | 20/28 | 18/28 |
| combo-r1 (actions + guards, one request) | 18/28 | |
r1 alone beats every cascade with holo. Its 8 failed runs: footcom (product page is a 404), eel and scott (results pages
show no product links in obscura), and Bunnings timber, where it called a 138x32 deck the product instead of giving up.

Held-out sites (83 tasks, 59 sites)
Round-1 tasks on sites held out by host hash, never trained on: far more varied stores than the test sets (office,
electronics, auto, garden, pets, farm...). The gain holds, smaller than on the test sets.

Table with columns: Kev alone, cascade | Kev alone | cascade |
|---|
| Kev prod | 27/83 (33%) | 39/83 (47%) |
| r1 | 36/83 (43%) | 37/83 (45%) |
| r1s (options shuffled + perm_kl) | 35/83 (42%) | 39/83 (47%) |
| combo-r1 (actions + guards, one request) | 34/83 (41%) | |
Caveat: these r1/r1s runs were served through a relay to molab; relay failures show up as kev_error escalations in
the cascade, so the cascade rows understate r1.
Offline: held-out sites, action accuracy
Steps from winning runs on sites held out by host hash. Caution: the harness orders options best-first, so
the right answer is the first option ~70% of the time; "always pick the first option" scores 66.7% here. The
shuffled rows (options reordered, keys renamed) remove that shortcut.

Table with columns: round-0 held-out sites (93 steps), harness order, shuffled, shuffled, answer not first| round-0 held-out sites (93 steps) | harness order | shuffled | shuffled, answer not first |
|---|
| always first option | 66.7% | 12.9% | 0% |
| Kev prod | 67.7% | 69.9% | 67.9% |
| r0b | 95.7% | 79.6% | 77.8% |
| r1 | 82.8% | 78.5% | 77.8% |
| r1s | 92.5% | 79.6% | 77.8% |
| combo-r1 final (actions + guards) | 94.6% | 77.4% | - |
r0b partly learned "pick the first option" (-16 points when shuffled); order-free, both rounds gain ~10 points
over prod. Gives up where a winning run acted: prod 13%, r0b/r1 0%. On 377 logged production requests r0b gives
up 9% vs prod 42%. Guard answers move on ~1% (devtools replay keeps them).
Training
Table with columns: r1 | r1 |
|---|
| init | jaredpalmer/kev-4b@139fdd94 (prod Kev), fresh each round (no compounding) |
| data | 1,395 action steps (round 0b + round 1) + 2,790 replayed evals/devtools-v1 records |
| epochs / records | 2 / 8,370 (10.1M tokens), max state 8,192 tokens, none truncated |
| LR | 2e-5 peak, OneCycle (10% warmup, cosine to 0), weight decay 0.01, head LR = LR |
| batch | 1 x accum 8 (1,048 optimizer steps), bf16, gradient checkpointing, LoRA rank 16 on all linear layers |
| grad norm | mean 9.0 / 9.7 per epoch, clipped on ~99% of steps |
| hardware | 1x RTX PRO 6000 Blackwell (96 GB): peak 23.6 GB GPU, 27.5 GB RAM |
| time | 65 min (0.47 s/record) |


Speed notes: flash-linear-attention = 2.5x (5.2 -> 2.1 s/record on long states); batch 1 x accum 8 is the
fastest setting; batch 8 + length sorting was 7x slower (padding). Collection, not training, is the bottleneck:
~2.5 h for 542 tasks x (Kev + cascade).
Serving: three adapters on one int8 base (zev-fast)
zev-rs branch multi-lora: Kev adapters on the same base share
one copy of it. The first one listed is merged (fused int8 path, full speed); the others attach as exact deltas
((W + dA) + (dX - dA) = W + dX). Measured on molab (RTX PRO 6000), int8, 377 requests at concurrency 16:
Table with columns: merged alone, on the shared 3-adapter server | merged alone | on the shared 3-adapter server |
|---|
| this adapter (listed first) | 27.5 req/s | 27.6 |
| kev-4b-guard@v2 | 27.7 | 22.8 |
| jxxxll/kev-4b-substitutes-judge | 152 | 129 |
| GPU memory | 9.8 GB each (29 GB for three) | 9.8 GB |

int8 vs bf16, merged vs unmerged: no measurable change on the held-out sets (actions 0.839-0.849 / 0.983-0.989, every
guard F1 0.987-0.988, within one or two examples). Full tables: llm-lab models/kev/README.md.
KEV_W8=int8 zev serve --model Flolep2607/kev-4b-actions@r1 --model Flolep2607/kev-4b-guard@v2 --model jxxxll/kev-4b-substitutes-judge# harness: HARNESS_KEV_URL=<server> HARNESS_KEV_GUARD_URL=<server> HARNESS_KEV_GUARD_MODEL=Flolep2607/kev-4b-guard@v2
Data: expert iteration (ReST-style RL)
Reward is reaching the target URL, never a model's opinion.
- Product URLs from the index (327 broad seed queries, marketplaces and test-set sites excluded), opened in
obscura through the harness; Claude Haiku labels product / not product / error and reads the product name.
- Tasks: start on the site's home, find the named product (round 0: 248 tasks / 167 sites; round 1: 542 / 373 new sites).
- Rollouts with every Kev request recorded (
HARNESS_KEV_RECORD=1): round 1 collected on-policy with r0b
(alone 173/542, cascade 198/542).
- Labels: steps of winning runs that changed the page, plus any step (from any run) where an option links to the
target page. Only the
action question is kept.
Data: Flolep2607/kev-rl-guard (actions/).
Code: flolep2607/llm-lab models/kev/finetune/harness-rl.
One adapter for actions + guards (combo-r1)
Trained on the action steps plus the guard pages (incl. reworded queries), 2x devtools replay. One request answers the
action and the three guards, so a done-check step costs one pass instead of two (zev int8: 36 ms vs 68 ms per step).
Table with columns: r1 + guard-v2 (2 requests), combo-r1 (1 request) | r1 + guard-v2 (2 requests) | combo-r1 (1 request) |
|---|
| test sets, exact checks | 20/28 | 18/28 |
| held-out sites | 36/83 | 34/83 |
| task_complete F1, held-out / reworded queries (zev int8) | 0.987 / 0.961 | 0.987 / 0.909 (3 false "done" vs 1) |
| action accuracy shuffled / log loss | 0.785 / 0.83 | 0.774 / 1.01 |
Cheaper, slightly weaker: the separate adapters stay the default; the combo needs more reworded-query guard data or a
separate guard head on its backbone.
Next
- r1-clean: r1 retrained on corrected labels (wins from the exact check, not the agent's claim), then benched.
- Round 2: difficulty filter (several sampled runs per task, keep the ones that sometimes pass) and a contrastive term
on Kev's choice probabilities.
- Retune the cascade's hand-off thresholds once r1 is served directly (no relay).
Use
python -m kev.serve --run <local snapshot of this repo> --port 8009# harness: HARNESS_KEV_URL=http://host:8009 HARNESS_KEV_GUARD_URL=<guard-v2 server>