Revisions
Table with columns: revision, checkpoint, what it is| revision | checkpoint | what it is |
|---|
main (also branch epoch-2) | checkpoint-98 | end of epoch 2, the final checkpoint |
epoch-1 (branch) | checkpoint-49 | end of epoch 1 |
Training
Trained on: the 1M-token slice of the base model's study notes (1,695 notes, 1,000,081 tokens; whole-note prefix of the seeded note order of notes_ntp_kl_20261012, so the 1M set is inside the 5M set, and so on up to 350M) plus the rollout notes, 6,602,566 tokens in all (391 packed rows per epoch), shuffled together. The rollout notes are notes that base Qwen3.5-9B wrote while looking back at its own graded solution attempts, without a reference solution. For each of 5,610 training implications (2,182 TRUE, 3,428 FALSE; none equals one of the 205 evaluation questions) the reading database held an index and eight base-model attempts, each with the score and the assessment of the judge model. It held no reference solution, no statement of the correct verdict and none of the equational-theory study sources (law pages, proved relations, methods): the writer saw the rollouts only. One session per problem; the writer chose what to open (read, glob, grep; at most 20 responses) and wrote one note, in a form of its own choosing. 5,609 notes, 5,602,485 training tokens (median 963 per note). Nothing checked a note against the correct verdict: by a regular-expression read about 8% of the notes conclude only the wrong verdict and about half mention the judge or a rubric. The judge's assessments were written against a rubric derived from the reference solution, so reference-derived wording can reach a note through them. The notes are not published.
Initialisation: Qwen/Qwen3.5-9B revision c202236235762e1c871ad0ccb60c8ee5ba337b9a (the released base model).
Recipe. Full-model next-token prediction on packed rows of 17,408 tokens (one row per GPU), 2 epochs,
learning rate 5e-06, cosine decay with 3% warmup, plus 0.01 x KL(base || student) on a frozen replay
pool of base-generated Equational trajectories (regularisation only, never a target). 8 x GH200, 98 optimizer
steps, 32 min of training, Slurm job 1146590 on TACC Vista, 2026-11-16. W&B run.
Table with columns: epoch 1 (epoch-1), epoch 2 (main) | epoch 1 (epoch-1) | epoch 2 (main) |
|---|
| checkpoint | checkpoint-49 | checkpoint-98 |
| mean train loss over the epoch (all training notes together) | 0.721 | 0.543 |
| held-out loss on the original study notes at the end of the epoch (base model: 1.245) | 0.429 | 0.413 |
| held-out loss on base-model trajectories (base model: 0.145) | 0.135 | 0.133 |
| KL(base || student) on the validation replay pool | 0.059 | 0.053 |
The held-out set is the frozen validation set of the recipe (original study notes and base-model trajectories); it contains
no rollout notes, so no number here measures the fit to unseen rollout notes.
What is in this repository
The safetensors are the serving export of the checkpoint (427 fine-tuned language tensors plus
348 unchanged base tensors; the text weights were fine-tuned, the vision components are the base
model's and were not evaluated for multimodal quality). The complete config, tokenizer, chat template and processors are included.
training_summary.json carries the provenance (launch record digest, checkpoint, tensor audit) and artifact_manifest.json the sha256
of every file.
Loading
from transformers import AutoModelForImageTextToText, AutoProcessor model_id = "U-WIN/qwen35-9b-equational-theory-sair-notes1m-noref-rollout-notes-ntp-kl-0p01"revision = "main" # "main" (= "epoch-2", final) or "epoch-1"processor = AutoProcessor.from_pretrained(model_id, revision=revision)model = AutoModelForImageTextToText.from_pretrained(model_id, revision=revision, dtype="auto", device_map="auto")
Code, prompts, note checks and the run log: recipe/equational_internalization/experiments/rollout_notes_noref_vista_20261116/README.md in the project repository.