Revisions
Table with columns: revision, checkpoint, what it is| revision | checkpoint | what it is |
|---|
main (also branch epoch-2) | checkpoint-378 | end of epoch 2, the final checkpoint |
epoch-1 (branch) | checkpoint-189 | end of epoch 1 |
Training
Trained on: the raw text of every file of the Equational study corpus of the project, the files that the base model could open with its reading tools when it wrote study notes: per-law pages (the law, its models, its relations, its lists of non-implications), relation pages (proved implications with their Lean proofs, dualities, equivalences, counterexamples) and exposition, foundations and methods pages. 53,467 files, 24,248,822 tokens, cut into 81,791 contiguous excerpts of about 512 tokens. Each excerpt is one training example: a Document: <path> header that is input only, then the excerpt text and an end-of-text token with loss, 24,330,613 loss tokens per epoch, every token of every file once per epoch (1,506 packed rows). The 1,552,542 non-implication pair pages that the corpus generates on request are not files and are not included. The corpus was built with the benchmark questions held out: its separation report (status PASS; exact ordered pairs, 1,100 evaluation targets) records that no protected implication is in the proved implications, in the non-implication index or named in the retained source text. Equivalent laws and shared tables remain, by design.
Initialisation: Qwen/Qwen3.5-9B revision c202236235762e1c871ad0ccb60c8ee5ba337b9a (the released base model).
Recipe. Full-model next-token prediction on packed rows of 17,408 tokens (one row per GPU), 2 epochs,
learning rate 5e-06, cosine decay with 3% warmup, plus 0.01 x KL(base || student) on a frozen replay
pool of base-generated Equational trajectories (regularisation only, never a target). 8 x GH200, 378 optimizer
steps, 57 min of training, Slurm job 1149071 on TACC Vista, started 2026-11-17. W&B run.
Table with columns: epoch 1 (epoch-1), epoch 2 (main) | epoch 1 (epoch-1) | epoch 2 (main) |
|---|
| checkpoint | checkpoint-189 | checkpoint-378 |
| mean train loss over the epoch (the run's own source text) | 0.177 | 0.084 |
| held-out loss on the original study notes at the end of the epoch (base model: 1.245) | 1.188 | 1.203 |
| held-out loss on base-model trajectories (base model: 0.145) | 0.146 | 0.146 |
| KL(base || student) on the validation replay pool | 0.018 | 0.015 |
The held-out set is the frozen validation set of the recipe (original study notes and base-model trajectories). It contains
none of this run's source text, so no number here measures the fit to unseen source text; the train loss of epoch 2 is on
text the model saw in epoch 1.
What is in this repository
The safetensors are the serving export of the checkpoint (427 fine-tuned language tensors plus
348 unchanged base tensors; the text weights were fine-tuned, the vision components are the base
model's and were not evaluated for multimodal quality). The complete config, tokenizer, chat template and processors are included.
training_summary.json carries the provenance (launch record digest, checkpoint, tensor audit) and artifact_manifest.json the sha256
of every file.
Loading
from transformers import AutoModelForImageTextToText, AutoProcessor model_id = "U-WIN/qwen35-9b-equational-theory-sair-study-materials-ntp-24m-kl-0p01"revision = "main" # "main" (= "epoch-2", final) or "epoch-1"processor = AutoProcessor.from_pretrained(model_id, revision=revision)model = AutoModelForImageTextToText.from_pretrained(model_id, revision=revision, dtype="auto", device_map="auto")
Code, data construction and the run log: recipe/equational_internalization/experiments/observations_study_materials_ntp_kl_vista_20261117/README.md in the project repository.