karuppusamym

slm-125m-legal-financial

README

License: apache-2.0

Model details

Table

Architecture	Llama (SwiGLU, RoPE, RMSNorm pre-norm, tied embeddings)
Parameters	125,847,552
Layers	12
Hidden size	768
Attention / KV heads	12 / 12
Intermediate size (SwiGLU)	3,072
Context length	1,024
Vocabulary	16,384 (byte-level BPE, trained from scratch on this corpus)
RoPE theta	10,000
Precision trained in	bf16 autocast, fp32 master weights

Training data

A 16,384-token byte-level BPE vocabulary and this model were both trained from scratch on a ~1.85B-token packed corpus (1,024-token windows), built from:

Table with columns: Source, Share of train tokens, Dataset
Source	Share of train tokens	Dataset
US case law	31.1%	`HFforLegal/case-law` (US split)
SEC filings	43.9%	`PleIAs/SEC`
Educational web text	25.0%	`HuggingFaceFW/fineweb-edu` (sample-10BT)

The corpus was cleaned (boilerplate/OCR/repetition/language filters), deduplicated (MinHash near-dup removal on case law, exact-dup removal everywhere), and decontaminated against the CaseHOLD and LexGLUE evaluation splits before packing. 645,388 documents in total; validation is a disjoint 1% held-out slice (18.7M tokens).

Training recipe

Table with columns: Setting, Value
Setting	Value
Objective	Next-token cross-entropy
Optimizer	AdamW, betas (0.9, 0.95), weight decay 0.1 (excluded on 1-D params)
Peak / floor learning rate	6e-4 / 6e-5
Warmup	200M tokens, then cosine decay
Precision	bf16 autocast
Gradient clipping	1.0 global norm
Global batch	524,288 tokens/optimizer step
Hardware	8× H100, single-node DDP
Schedule (planned)

Validation perplexity trajectory

Full validation split, evaluated every 1,000 optimizer steps (perplexity = exp(mean token-weighted cross-entropy)):

Table with columns: Step, Val loss, Val perplexity
Step	Val loss	Val perplexity
1,000	2.777	16.07
2,000	2.524	12.48
3,000	2.421	11.26
4,000	2.361	10.60
5,000	2.319	10.17
6,000	2.286	9.83

Training loss fell from 9.87 (random init) to ~2.17 over the same span.

Sample completions

Generated with do_sample=True, temperature=0.8, top_p=0.95 at this checkpoint. These are illustrative, not curated — a base model can produce incoherent, biased, or fabricated legal/financial content, and nothing it generates should be treated as legal or financial advice.

Prompt: "The court held that" Continuation: "...the defendant was not prejudiced by the admission of the testimony relating to the statement that he was not under arrest. In so holding, the court stated: We agree. In State v. Hensley..."

Prompt: "Net revenues for the fiscal year increased" Continuation: "... $2.5 mi l l i o n, or 7.1$ 31.9 million in fiscal 1997 from $31.9 million in fiscal 1996. The increase in total revenues was attributable to the Company..."

Intended use & limitations

Base completer only. No instruction tuning, no RLHF, no chat template training — <|user|>/<|assistant|>/<|system|> tokens are reserved in the tokenizer for a future instruction-tuning phase but this checkpoint was never trained on turn-structured data.
Not a source of legal or financial advice or fact. It will produce fluent, plausible-sounding legal and financial prose, including plausible case names, statutes, and figures that may be entirely fabricated.
Partially trained, as noted above — expect rougher edges than a completed 5-epoch run.
Small model, narrow domain. 125M parameters trained on ~1.85B tokens is small by current standards; general-knowledge and reasoning capability outside the legal/financial/web-text domain is limited.

How to use

python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "karuppusamym/slm-125m-legal-financial"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

inputs = tok("The court held that", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=60, do_sample=True, temperature=0.8, top_p=0.95)
print(tok.decode(out[0], skip_special_tokens=True))

License

Model weights released under Apache 2.0. Training data provenance: US case law is public-domain court opinion text; SEC filings are public regulatory filings; FineWeb-Edu is a permissively-licensed educational web-text dataset. This repository does not redistribute the training data itself.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

karuppusamym

Model Tree

Base

this model

Input Modalities

Text

Output Modalities