Training data
- Source:
pythonformer/agents-learn-runtime-train,
data/knapsack/stateless/traces.jsonl — 1000 CodeAct agent traces generated under the
stateless runtime, teacher model Gemini (see paper Table 1, 17.8-22.8% teacher exact-solve rate).
- Copied byte-identical (sha256-verified) into the training pipeline, zero transformation.
- Not a re-derived/reconstructed trace set — this repo's earlier attempts to regenerate an
equivalent set locally (
prepare_paired_data.py, 1109 pairs) were found to NOT match this
canonical data and were discarded; see HANDOFF_knapsack_seed_sweep.md in the repo for the
full story.
Hyperparameters
Table | |
|---|
| Base model | Qwen/Qwen3-8B |
| Quantization | 4-bit QLoRA (bitsandbytes NF4) |
| LoRA rank / alpha / dropout | 64 / 128 / 0.05 |
| LoRA target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Training context | 16384 tokens |
| Batch size | 1 (micro) x 16 (grad accum) = 16 effective |
| Epochs | 3 (~45 optimizer steps over 1000 examples) |
| Optimizer / LR / schedule | AdamW (torch) / 1e-4 / cosine, 3% warmup |
| Training seed | 1337 |
| Thinking | disabled (enable_thinking: false, i.e. /nothink) |
Final training loss: 0.1313 (this run) — manifest and full Axolotl config recorded in
manifest.json / axolotl_config.yaml in this repo, also mirrored into this HF repo for
provenance.
Benchmark results (this checkpoint: stateless-trained, seed 1337)
n=300 tasks/cell, from
pythonformer/agents-learn-runtime-tasks
(easy/medium knapsack splits). "matched" = eval runtime equals the training runtime; "cross-eval"
= eval runtime differs from training runtime (tests the amnesia-tax hypothesis). Metric is
score (metrics.score.mean, partial-credit normalized optimality = agent's basket value /
optimal basket value, 0-1) — used instead of the stricter exact-optimum-match rate
(average_is_solved), which is too sparse at n=300 for this task to compare cleanly. Three
training seeds shown side by side, no averaging — seed 3407 is the initial run (source:
pythonformer/agents-learn-runtime-benchmarks,
gated dataset, request access on the dataset page, n=300/cell), seed 1337 / seed 777 are
this variance sweep.
Table with columns: eval runtime, difficulty, seed 3407 (initial), seed 1337, seed 777| eval runtime | difficulty | seed 3407 (initial) | seed 1337 | seed 777 |
|---|
| persistent (cross-eval) | easy | 0.294 | 0.143 | 0.155 |
| persistent (cross-eval) | medium | 0.092 | 0.015 | 0.032 |
| stateless (matched) | easy | 0.614 | 0.137 | 0.117 |
Seed 3407 numbers use the dataset's folder naming {runtime}_{model}_knapsack_{difficulty}
— confirmed by downloading a cell's config.yaml and reading its model:/persistent_state:
fields directly (the dataset's own README states the opposite order, {model}_{runtime},
but that does not match this snapshot's actual folder structure).
Base model reference (seed-independent, not this checkpoint)
Table with columns: eval runtime, difficulty, seed 3407 (initial), seed 1337 / seed 777 (base, seed-independent)| eval runtime | difficulty | seed 3407 (initial) | seed 1337 / seed 777 (base, seed-independent) |
|---|
| persistent | easy | 0.050 | 0.061 |
| persistent | medium | 0.030 | 0.030 |
| stateless | easy | 0.034 | 0.035 |
| stateless | medium | 0.030 | 0.017 |
Citation
@article{pythonformer260301209,
title = {Agents Learn Their Runtime: Interpreter Persistence as Training-Time Semantics},
author = {Victor May, Aaditya Salgarkar, Yishan Wang, Diganta Misra, Huu Nguyen},
journal = {arXiv preprint arXiv:2603.01209},
year = {2026}
}