Two revisions — pick deliberately
Table with columns: Revision, Data, Judge /5, Forgetting vs base, Notes| Revision | Data | Judge /5 | Forgetting vs base | Notes |
|---|
main | 2,000 pairs | 1.50 | +9.5% | The better model. Recommended. |
day-10 | 10,000 pairs | 1.54 | +16.3% | Study endpoint. 5× the data, no quality gain, ~2× the forgetting, and a regressed probe answer. |
The two judge scores are identical within the measured ±0.07 noise band, so main (day 2)
strictly dominates: same quality, half the damage, one-fifth the data.
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Ace-2504/fine-tuned-125m-slm")
model = AutoModelForCausalLM.from_pretrained("Ace-2504/fine-tuned-125m-slm")
tok10 = AutoTokenizer.from_pretrained("Ace-2504/fine-tuned-125m-slm", revision="day-10")
model10 = AutoModelForCausalLM.from_pretrained("Ace-2504/fine-tuned-125m-slm", revision="day-10")
msgs = [
{"role": "system", "content": "You are a precise legal and financial assistant."},
{"role": "user", "content": "What is the standard of proof in a civil lawsuit?"},
]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")
print(tok.decode(model.generate(ids, max_new_tokens=120)[0][ids.shape[1]:], skip_special_tokens=True))
Honest limitations
- Answer quality is poor (~1.5/5). It learned the shape of an assistant turn, not reliable
answers. Expect repetition, prompt echoing, and confident wrong answers.
- It forgot part of its pretraining. Perplexity on the original pretraining validation set rose
from 11.35 → 12.42 (
main) / 13.20 (day-10).
- Root cause: ~40% of training was closed-book QA — recalling a document fact from a single
exposure, which a 125M model cannot do. Common facts land; long-tail specifics never do.
- Do not use for legal or financial advice. It is a study artifact.
Training
Full fine-tune (no LoRA), 3 epochs, batch 16, LR 3e-5→3e-6 cosine, AdamW, bf16, assistant-only loss
masking, seed 1337, on a single Modal L4. Data: QnA written by gemini-3.1-flash-lite, grounded in
passages from the same corpus the base was pretrained on, then filtered, deduplicated (exact +
embedding), decontaminated and balanced. Chat template uses <|system|> / <|user|> / <|assistant|>.
generation_config ships with repetition_penalty=1.3 and no_repeat_ngram_size=3 — without them
greedy decoding falls into repetition loops.