Revisions
Table with columns: revision, checkpoint, what it is| revision | checkpoint | what it is |
|---|
main (also branch epoch-2) | checkpoint-224 | end of epoch 2, the final checkpoint |
epoch-1 (branch) | checkpoint-112 | end of epoch 1 |
Training
Trained on: the 10M-token slice of the base model's study notes (16,737 notes, 10,000,050 tokens; whole-note prefix of the seeded note order of notes_ntp_kl_20261012, so the 1M set is inside the 5M set, and so on up to 350M) plus the rollout notes, 15,072,278 tokens in all (889 packed rows per epoch), shuffled together. The rollout notes are notes that base Qwen3.5-9B wrote while looking back at its own graded solution attempts, without a reference solution but with the equational-theory study sources. For each of 5,610 training implications (2,182 TRUE, 3,428 FALSE; none equals one of the 205 evaluation questions) the reading database held an index and eight base-model attempts, each with the score of the judge model but not the judge's written assessment, and next to them the equational-theory study sources of the project: law pages, the proved implications with their Lean proofs, dualities, exposition, foundations and methods pages. It held no reference solution and no statement of the correct verdict. The sources do contain, for every TRUE problem, the proved implication itself with its Lean proof (opened in 94% of the TRUE sessions); they contain no page about any FALSE problem. One session per problem; the writer chose what to open (read, glob, grep; at most 20 responses) and wrote one note, in a form of its own choosing. 5,602 notes, 5,072,228 training tokens (median 871 per note). Nothing checked a note against the correct verdict: by a regular-expression read about 14% of the notes conclude only the wrong verdict (most of them on FALSE problems between higher-order laws), and an exact check of the operations written in the notes finds a working counterexample in 674 of the 3,421 notes on FALSE problems (a lower bound). About 12% of the notes mention the judge. The notes are not published.
Initialisation: Qwen/Qwen3.5-9B revision c202236235762e1c871ad0ccb60c8ee5ba337b9a (the released base model).
Recipe. Full-model next-token prediction on packed rows of 17,408 tokens (one row per GPU), 2 epochs,
learning rate 5e-06, cosine decay with 3% warmup, plus 0.01 x KL(base || student) on a frozen replay
pool of base-generated Equational trajectories (regularisation only, never a target). 8 x GH200, 224 optimizer
steps, 44 min of training, Slurm job 1147837 on TACC Vista, started 2026-11-16. W&B run.
Table with columns: epoch 1 (epoch-1), epoch 2 (main) | epoch 1 (epoch-1) | epoch 2 (main) |
|---|
| checkpoint | checkpoint-112 | checkpoint-224 |
| mean train loss over the epoch (all training notes together) | 0.523 | 0.379 |
| held-out loss on the original study notes at the end of the epoch (base model: 1.245) | 0.339 | 0.333 |
| held-out loss on base-model trajectories (base model: 0.145) | 0.128 | 0.127 |
| KL(base || student) on the validation replay pool | 0.060 | 0.053 |
The held-out set is the frozen validation set of the recipe (original study notes and base-model trajectories); it contains
no rollout notes, so no number here measures the fit to unseen rollout notes.
What is in this repository
The safetensors are the serving export of the checkpoint (427 fine-tuned language tensors plus
348 unchanged base tensors; the text weights were fine-tuned, the vision components are the base
model's and were not evaluated for multimodal quality). The complete config, tokenizer, chat template and processors are included.
training_summary.json carries the provenance (launch record digest, checkpoint, tensor audit) and artifact_manifest.json the sha256
of every file.
Loading
from transformers import AutoModelForImageTextToText, AutoProcessor model_id = "U-WIN/qwen35-9b-equational-theory-sair-notes10m-noref-src-rollout-notes-ntp-kl-0p01"revision = "main" # "main" (= "epoch-2", final) or "epoch-1"processor = AutoProcessor.from_pretrained(model_id, revision=revision)model = AutoModelForImageTextToText.from_pretrained(model_id, revision=revision, dtype="auto", device_map="auto")
Code, prompts, note checks and the run log: recipe/equational_internalization/experiments/rollout_notes_noref_sources_vista_20261116/README.md in the project repository.