What it does
The base model, given "In a breach of contract claim, the plaintiff…", would ramble
onward. This model, asked a question, answers it:
Q: In a breach of contract claim, what must the plaintiff prove?
A: The plaintiff must show: (1) a contract; (2) an agreement to perform a
specific act; (3) an obligation to perform…
The tokenizer has role special tokens but no chat template, so format prompts
manually as <|bos|><|system|>{system}<|user|>{question}<|assistant|> and let the
model complete the answer up to <|eos|>:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("jonam-ai/legal-slm-125m-sft")
model = AutoModelForCausalLM.from_pretrained("jonam-ai/legal-slm-125m-sft", torch_dtype=torch.bfloat16)
system = "You are a knowledgeable legal and financial assistant. Answer accurately and concisely."
question = "What is the purpose of a Form 10-K filing?"
def sid(t): return tok.convert_tokens_to_ids(t)
ids = (tok("<|bos|>", add_special_tokens=False)["input_ids"]
+ [sid("<|system|>")] + tok(system, add_special_tokens=False)["input_ids"]
+ [sid("<|user|>")] + tok(question, add_special_tokens=False)["input_ids"]
+ [sid("<|assistant|>")])
out = model.generate(torch.tensor([ids]), max_new_tokens=120, do_sample=True,
temperature=0.7, top_p=0.9, eos_token_id=sid("<|eos|>"),
pad_token_id=sid("<|pad|>"))
print(tok.decode(out[0][len(ids):], skip_special_tokens=True))
Training data
5,846 grounded question–answer pairs, synthesized with a teacher-LLM
distillation pipeline over a cleaned corpus of US case law, SEC filings, and
educational web text:
- Chunk the corpus into ~800-token passages.
- Generate (Gemini Flash-Lite): write self-contained Q&A answerable only
from the passage, balanced across task types (QA, extraction, summarization,
rewrite) and difficulty (easy → hard).
- Judge (Gemini Flash as LLM-as-judge): keep only pairs that are grounded,
correct, and self-contained (score ≥ 4/5). ~78% pass.
- Dedup (exact + MinHash-LSH near-duplicate removal).
- Format as chat and tokenize with the base model's own tokenizer, with
loss computed only on the answer tokens.
Mix: case-law 45% · SEC 45% · web 10%. Split: 5,554 train / 292 val.
Training
Table | |
|---|
| Method | full fine-tune (not LoRA) |
| Hardware | 1 × NVIDIA L4 |
| Epochs | 2 |
| Tokens seen | ~1.0M |
| LR | 3e-5, cosine decay, 3% warmup |
| Precision | bf16 compute, fp32 master weights |
| Time | ~80 seconds |
Limitations
125M parameters, English only, 1,024-token context, not aligned/RLHF'd. It imitates
the form of grounded answers learned from a synthetic dataset; factual reliability
is limited by model size. Not legal or financial advice.