karnan

slm125m-legal-base

README

License: apache-2.0

Try it

👉 Run it in Colab — free CPU, no GPU needed, loads in seconds.

Or locally — the model is ~500MB (fp32) and runs on CPU:

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "karnan/slm125m-legal-base"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = "In determining whether the search was reasonable, the court"
ids = tok(prompt, return_tensors="pt")["input_ids"]   # note: input_ids only

out = model.generate(
    input_ids=ids,
    max_new_tokens=80,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
    repetition_penalty=1.1,
)
print(tok.decode(out[0], skip_special_tokens=True))

Note: pass input_ids explicitly. This tokenizer also emits token_type_ids, which LlamaForCausalLM.generate() rejects. **tok(prompt) will raise a ValueError.

Give it a prefix, not a question. "What is a tort?" will get you a rambling continuation; "A tort is" will get you a definition.

What it actually produces

Fourth Amendment analysis — correct doctrinal framing, fabricated quote:

In determining whether the search was reasonable, the court must assess its effect on the intrusion on an individual's privacy. The United States Supreme Court has stated that the exclusionary rule of the Fourth Amendment "does not require that the evidence be excluded at all." Coolidge v. New Hampshire, 403 U.S. 443, 454 (1971)...

Enumerated civil pleading — genuinely well-formed legal reasoning:

The plaintiff alleges that the defendant 's failure to comply with this provision of the contract made the building of the plaintiff's property and premises impossible, because (1) the defendant was without authority to enter into a lease for any purpose, (2) the plaintiff had no adequate remedy at law; (3) the defendant breached the implied covenants in the lease by failing to maintain the common...

Contract / default clause — correct structure and register:

Pursuant to the terms of this Agreement, upon the occurrence of an Event of Default, General Partner shall have the option to: (i) pay or commence a voluntary case under Chapter 11 of the Bankruptcy Code; (ii) make all payments required to be made in accordance with the payment terms of the Notes...

SEC-style MD&A — fluent prose, wrong arithmetic:

The Company's net revenues for the fiscal year ended December 31, 1996 increased to $487.2 mi l l i o n f r o m$ 393.9 million, an increase of $125.0 million or 55% over the comparable period of 1995...

Note it switches register correctly between legal and financial prompts — the domain training worked. The facts did not.

Model

Table

Parameters	125,847,552
Architecture	Llama (RMSNorm, SwiGLU, RoPE, MHA)
Layers / hidden / heads	12 / 768 / 12 (head dim 64)
Context length	1,024
Vocab	16,384 — byte-level BPE trained on this corpus
Embeddings	Tied
Precision	bf16 autocast, fp32 master weights

The tokenizer is domain-trained, not borrowed. "Petition for a writ of certiorari to the United States Court of Appeals is denied" encodes to 17 tokens — a general-purpose tokenizer spends several on certiorari alone. Byte-level means no UNK tokens ever.

Training data

2.04B tokens, built from three public datasets:

Table with columns: Source, Tokens, Share
Source	Tokens	Share
SEC filings (`PleIAs/SEC`)	860M	42%
US case law (`HFforLegal/case-law`)	715M	35%
FineWeb-Edu (fluency filler)	464M	23%
Total	2.04B	77% legal

The mix is legal-first, not 70/20/10. The legal sources are supply-bound — measured, they cap at ~2B tokens combined — so a 70% case-law corpus at 10B tokens is arithmetically impossible. We take all of both legal sources and cap the web filler at 0.5B. The web slice is not padding: a pure-legal corpus produces a model that has never seen ordinary English.

Pipeline: 6-step deterministic cleaning → MinHash-LSH near-dedup → exact dedup → decontamination against CaseHOLD/LexGLUE (23,995 documents removed) → 16K BPE → packed uint16 1024-token windows, 99/1 split.

The decontamination matters: without it, the perplexity below would be measuring memorization of the eval set rather than learning.

Training

5 epochs (10.20B tokens seen), 2×H100, 19,448 steps, ~4 hours, $29.22. AdamW (β 0.9/0.95, wd 0.1), cosine 6e-4 → 6e-5, 200M-token warmup, 0.5M-token batches.

Table with columns: step, val ppl, step, val ppl, step, val ppl
step	val ppl	step	val ppl	step	val ppl
1,000	18.72	8,000	10.59	15,000	9.45
2,000	14.43	9,000	10.35	16,000	9.36
3,000	12.93	10,000

Perplexity is the only metric reported. At 125M parameters, MMLU-style benchmarks are near-random noise; publishing them would be theatre.

Cost

Table with columns: Phase, Cost
Phase	Cost
Data (clean, dedup, tokenizer, pack — all CPU)	~$1
GPU benchmark + resume drill	~$0.48
Pretraining (2×H100, 5 epochs)	$29.22
Total	~$31

A prior implementation of this same project cost $56.58 for the identical 5 epochs. The ~2× reduction came from torch.compile + flash attention (SDPA) + fused AdamW + TF32 — measured, by benchmarking both an H100 (420K tok/s) and an A100-80GB (173K tok/s) before spending anything. The H100 turned out to be cheaper per token despite costing 1.58× more per hour, because it is 2.4× faster.

Two bugs worth knowing about

1. The OCR filter was deleting 58% of the corpus. The obvious threshold — drop a document if >20% of its words are absent from /usr/share/dict/words — is catastrophic for legal text. That dictionary is a spelling dictionary: no inflections (alleges), no legal register (hereto), no Latin (certiorari), no citations. Clean legal prose scores a median 0.21 against it. The 0.20 cutoff sat directly on the mode of the distribution. Real OCR garbage scores ~0.96 — the populations are trivially separable, and 0.20 was simply on the wrong side of the wrong one. Corrected to 0.35.

2. The checkpoint-resume path was broken. torch.save/load round-trips the RNG state to a non-ByteTensor, so torch.set_rng_state raised on every resume. Modal preempts every GPU function by default — there is no non-preemptible tier at any price — so every preemption would have killed the run. Caught by a deliberate $0.20 kill/resume drill before committing to a 4-hour unattended run.

An honest negative result

The prior implementation reached val ppl 8.50. This model reached 9.20 — we lost by 0.70.

I had predicted the opposite. I attributed the prior run's late-curve flatline (8.62 → 8.50 over its final 6,000 steps) to its use of bf16 master weights, which round small updates to zero once the learning rate decays. That mechanism is real — but I inferred causation from a curve shape and never tested the null hypothesis, which is simply that a converged cosine schedule flattens at its tail.

The likely actual cause of the gap: this corpus is 7% smaller (2.04B vs 2.19B tokens), because the domain tokenizer compresses legal text better. At the same 5 epochs, this run saw 10.20B tokens against the prior run's 10.95B. Fewer tokens seen → worse perplexity.

The irony: the tokenizer working better is part of why the headline number looks worse, because holding epochs constant does not hold tokens seen constant.

Reproduce it

Every phase is one worker per data shard (Modal preempts by default, so a preemption costs one shard, not the run):

bash
modal run modal_app.py::clean       # stream + clean       ~15 min
modal run modal_app.py::dedup       # dedup + decontam     ~6 min
modal run modal_app.py::tokenizer   # 16K BPE              ~4 min
modal run modal_app.py::tokenize    # pack uint16 windows  ~10 min
modal run modal_app.py::drill       # GATE: prove resume works
modal run modal_app.py::benchmark   # measure $/1B-tokens per GPU
modal run modal_app.py::pretrain --gpu H100 --max-usd 40

License

Apache 2.0. Training data is public: HFforLegal/case-law, PleIAs/SEC, HuggingFaceFW/fineweb-edu — consult each for its own terms.

Not legal advice. Not a source of legal facts.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.