Revisions
Table with columns: revision, checkpoint, what it is| revision | checkpoint | what it is |
|---|
main (also branch epoch-2) | checkpoint-164 | end of epoch 2, the final checkpoint |
epoch-1 (branch) | checkpoint-82 | end of epoch 1 |
Training
Trained on: what the reading tools returned to base Qwen3.5-9B while it explored the Equational study corpus to write study notes: the returns of read, glob and grep that have content, from the exploration sessions of rounds R9 to R14 of the note collection (409,922 sessions, 751.6M tokens of returned text in all). Each return is one training example: a header naming the call (for example read relations/dualities/Eq4463-Eq4503/proof.md lines 1-22 of 22) is input only, then the returned text and an end-of-text token with loss. The model's own text, the task prompts and every return of the note tools (searching, reading, writing and checking notes) are not in it. A return is kept as often as it occurred, so a page that many sessions read is in the set many times. Sessions are in a seeded order and each scale is a whole-example prefix of that order, so the 1M set is inside the 5M set, and so on up to 350M. This run: 9,999,944 loss tokens from 44,629 returns of 5,410 sessions (9,128,591 tokens from read, 855,231 from glob, 16,122 from grep; 656 packed rows per epoch). Counting each distinct returned text once leaves 6,478,192 of the 9,955,315 content tokens (65%). The corpus was built with the benchmark questions held out: its separation report (status PASS; exact ordered pairs, 1,100 evaluation targets) records that no protected implication is in the proved implications, in the non-implication index or named in the retained source text. Equivalent laws and shared tables remain, by design.
Initialisation: Qwen/Qwen3.5-9B revision c202236235762e1c871ad0ccb60c8ee5ba337b9a (the released base model).
Recipe. Full-model next-token prediction on packed rows of 17,408 tokens (one row per GPU), 2 epochs,
learning rate 5e-06, cosine decay with 3% warmup, plus 0.01 x KL(base || student) on a frozen replay
pool of base-generated Equational trajectories (regularisation only, never a target). 8 x GH200, 164 optimizer
steps, 39 min of training, Slurm job 1149068 on TACC Vista, started 2026-11-17. W&B run.
Table with columns: epoch 1 (epoch-1), epoch 2 (main) | epoch 1 (epoch-1) | epoch 2 (main) |
|---|
| checkpoint | checkpoint-82 | checkpoint-164 |
| mean train loss over the epoch (the run's own source text) | 0.197 | 0.057 |
| held-out loss on the original study notes at the end of the epoch (base model: 1.245) | 1.193 | 1.212 |
| held-out loss on base-model trajectories (base model: 0.145) | 0.148 | 0.147 |
| KL(base || student) on the validation replay pool | 0.021 | 0.018 |
The held-out set is the frozen validation set of the recipe (original study notes and base-model trajectories). It contains
none of this run's source text, so no number here measures the fit to unseen source text; the train loss of epoch 2 is on
text the model saw in epoch 1.
What is in this repository
The safetensors are the serving export of the checkpoint (427 fine-tuned language tensors plus
348 unchanged base tensors; the text weights were fine-tuned, the vision components are the base
model's and were not evaluated for multimodal quality). The complete config, tokenizer, chat template and processors are included.
training_summary.json carries the provenance (launch record digest, checkpoint, tensor audit) and artifact_manifest.json the sha256
of every file.
Loading
from transformers import AutoModelForImageTextToText, AutoProcessor model_id = "U-WIN/qwen35-9b-equational-theory-sair-observations-ntp-10m-kl-0p01"revision = "main" # "main" (= "epoch-2", final) or "epoch-1"processor = AutoProcessor.from_pretrained(model_id, revision=revision)model = AutoModelForImageTextToText.from_pretrained(model_id, revision=revision, dtype="auto", device_map="auto")
Code, data construction and the run log: recipe/equational_internalization/experiments/observations_study_materials_ntp_kl_vista_20261117/README.md in the project repository.