DeependraVerma

legal-slm-125m-sft

What this is

The base model only continues text — it cannot answer a question. This model answers questions in two distinct modes, and they are not equally reliable:

Open-book (recommended) — you provide a contract excerpt or passage in the prompt along with your question, and the model extracts/classifies/ answers from that given text. This is the mode this revision specifically strengthens, and it is measurably the more trustworthy one (see Evaluation).
Closed-book — you ask a general legal/financial question with no source text, and the model answers from whatever it retained during training. This mode is measurably weaker and prone to confident fabrication (see below) — it is a known, structural limitation of a 125M model recalling specific, often single-occurrence facts from a 2B-token pretraining corpus, not something SFT data changes can fix.

Model description

Same architecture and parameter count as the base model — full fine-tuning does not change the model's shape, only its weights:

Table

Parameters	125,848,320 (~125.8M, tied embeddings)
Architecture	Llama-style decoder, 12 layers / 768 hidden / 12 heads (head dim 64, full MHA)
MLP	SwiGLU, intermediate size 3,072
Positional encoding	RoPE, θ = 10,000
Vocabulary	16,384 byte-level BPE (same tokenizer as the base model, plus role tokens)
Context length	1,024 tokens
Precision	bf16 compute · fp32 master weights and saved checkpoint
Base checkpoint	`DeependraVerma/slm-125m-base` (this author's own from-scratch pretrain, not a third-party checkpoint)

Intended uses & limitations

Intended use: open-book contract clause extraction and classification (paste an excerpt, ask what a clause covers or how it should be categorized); general short-form legal/financial QA, summarization, and rewriting as a secondary, weaker capability. Research, education, and experimentation with small-model instruction tuning.

Not intended for: any use where factual accuracy is required, or as a substitute for a lawyer, accountant, or financial advisor. At 125M parameters, this model has hard, documented limits — see below.

Known limitations — read before using

A full held-out evaluation (all validation examples, not a spot-check — see Evaluation below) plus manual inspection of failure cases found:

Open-book contract tasks work reasonably well. Given a contract excerpt, the model extracts the relevant clause (or correctly says a clause type isn't present) most of the time, and classifies clause categories correctly roughly 7 times out of 10.
Closed-book general Q&A (no source text given) is weak and prone to confident, specific fabrication. Manual review of failed held-out examples found the model inventing plausible-but-wrong specifics — e.g., asked what a doctor's diagnosis was in a case excerpt, it confidently stated a specific but entirely fabricated clinical detail that contradicted the real answer; asked about a case holding, it invented a different, incorrect legal conclusion. This is not evasive or hedged — it states invented specifics with the same confidence as correct ones, which is the actual risk with a model this size doing closed-book fact recall.
This is a capacity limitation, not a data quality issue. A controlled test (retraining on the original, smaller closed-book-only dataset and re-measuring with the same methodology) showed the same weakness was already present before this revision's data additions — it reflects what a 2-epoch, 2B-token pretrain at this parameter count can and cannot retain about specific, often single-occurrence facts, not a defect introduced by any one training run.

Never use this model's output as legal, financial, or factual advice. Always treat specific claims (numbers, names, citations, case holdings) as unverified until checked against a primary source — especially in closed-book use (no excerpt provided).

Training data

Two complementary sources of supervision:

Closed-book pairs — via teacher-LLM distillation, entirely on-prem at zero external API cost: a locally hosted Meta-Llama-3.1-70B-Instruct (served via vLLM) generated grounded Q&A from ~800-token passages of the pretraining corpus, judged by a mix of NVIDIA Nemotron-Ultra-550B and the same Llama-70B teacher (score ≥ 4/5 kept). This revision reattaches each pair's original source passage into the prompt (open-book), which is what improved this task family's correctness (see Evaluation) — the same underlying teacher-generated Q&A as before, just no longer trained to answer without seeing its source.

Open-book contract pairs (new this revision) — two public, CC BY 4.0 datasets, used as ground truth directly (no LLM teacher, no synthetic generation):

theatticusproject/cuad-qa — expert-annotated commercial contract clause extraction, 41 clause categories across hundreds of real EDGAR-filed contracts. Answer excerpts are windowed around the true answer span (not truncated from the top of the document) so every training example is actually grounded in what the model is shown.
coastalcph/lex_glue (ledgar config) — labeled contract provisions, 100 legal clause categories, SEC contract-exhibit sourced.

Table

Final curated pairs	21,186 (20,127 train / 1,059 val)
Source mix	CUAD (contracts) ~51% · LEDGAR (contracts) ~22% · SEC filings ~13% · US case law ~12% · educational web ~2.5%
Task mix	extraction ~60% · classification ~22% · QA ~14% · summarization ~5% · rewrite <1%
Dedup / decontamination	exact + MinHash-LSH near-dup removal, scoped per source document so identical question templates across different contracts aren't collapsed together; disjoint train/val split by construction

Training procedure

Table

Method	full fine-tune (not LoRA), from the base checkpoint
Hardware	1 GPU (this project's on-prem 8×B200 box)
Epochs	2
Learning rate	3e-5, cosine decay, 3% warmup
Precision	bf16 compute, fp32 master weights
Loss masking	answer tokens only — prompt (system + user turns, including any contract excerpt) masked to `-100`
Wall-clock time	~166 seconds

Chat format — same as the base project's convention, with an optional excerpt folded into the user turn for open-book examples:

markdown
<|bos|><|system|>{system}<|user|>{excerpt (if any)}\n\nQuestion: {question}<|assistant|>{answer}<|eos|>

Default system prompt: "You are a knowledgeable legal and financial assistant. Answer accurately and concisely."

Evaluation

Full held-out validation set (1,080 examples, not a sample), scored per task family since they're genuinely different shapes — extraction/classification against ground truth, general QA judged by an independent local Meta-Llama-3.1-70B-Instruct for semantic correctness against the reference answer:

Table with columns: Task, Metric, Result
Task	Metric	Result
CUAD contract clause extraction (open-book)	Token-F1	0.711
CUAD "clause not present" refusal (open-book)	Accuracy	90.1%
LEDGAR clause classification (open-book), 100 categories	Exact-match accuracy	74.1%
Case law general Q&A (mostly open-book after this revision)	Llama-70B-judged correctness	35.6%
SEC filings general Q&A (mostly open-book after this revision)	Llama-70B-judged correctness	43.1%

The general-QA numbers are a large, measured improvement over the same methodology applied to the pre-revision model (8.1% / 16.9% / 3.0% respectively) — but manual review of the remaining failures found real confident fabrication, not just imprecision (see Known limitations). Treat the contract-extraction numbers as the more trustworthy capability of this model.

A specific bug found and fixed after initial release of this revision: a handful of clause categories exist by the same name in both CUAD (extraction, "quote the text") and LEDGAR (classification, "name the category") — e.g. Governing Law, Effective Date, Insurance, Non-Disparagement. Training on both caused the model to collapse to just outputting the bare category name for those specific clause types even when asked to extract/quote, regardless of question phrasing — confirmed on real held-out contracts and via hand-written test excerpts, 100% reproducion rate before the fix. Fixed by removing the ~400 conflicting LEDGAR examples for those categories and reformatting all LEDGAR answers to "This clause is categorized as: X" instead of a bare label, so classification and extraction outputs are structurally distinguishable everywhere, not just for the categories found. The numbers above are post-fix.

How to use

Open-book (recommended) — provide the excerpt in the prompt:

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("DeependraVerma/legal-slm-125m-sft")
model = AutoModelForCausalLM.from_pretrained(
    "DeependraVerma/legal-slm-125m-sft", torch_dtype=torch.bfloat16
)

system = "You are a knowledgeable legal and financial assistant. Answer accurately and concisely."
excerpt = "This Agreement may be terminated by either party upon 30 days written notice..."
question = f"{excerpt}\n\nQuestion: What are the termination terms of this agreement?"

def sid(t):
    return tok.convert_tokens_to_ids(t)

ids = (
    tok("<|bos|>", add_special_tokens=False)["input_ids"]
    + [sid("<|system|>")] + tok(system, add_special_tokens=False)["input_ids"]
    + [sid("<|user|>")] + tok(question, add_special_tokens=False)["input_ids"]
    + [sid("<|assistant|>")]
)
out = model.generate(
    torch.tensor([ids]), max_new_tokens=120, do_sample=True,
    temperature=0.7, top_p=0.9, eos_token_id=sid("<|eos|>"), pad_token_id=sid("<|pad|>"),
)
print(tok.decode(out[0][len(ids):], skip_special_tokens=True))

For browser/edge inference (transformers.js, no server), see DeependraVerma/legal-slm-125m-sft-onnx — note that export predates this revision and has not yet been regenerated from these weights.

Citation

bibtex
@misc{verma2026legalslm125msft,
  author = {Deependra Verma},
  title  = {legal-slm-125m-sft: A Fine-Tuned Q\&A Assistant on a From-Scratch 125M Legal/Financial Language Model},
  year   = {2026},
  url    = {https://huggingface.co/DeependraVerma/legal-slm-125m-sft},
  note   = {Code: https://github.com/DeependraVerma/legal-slm-125M}
}

Author

Deependra Verma — Generative AI Researcher / AI Engineer. GitHub · Hugging Face

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

DeependraVerma

Model Tree

Base

DeependraVerma/slm-125m-base

Fine-tuned

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Container

Explore FriendliAI today

Get started Talk to an engineer

What this is

The base model only continues text — it cannot answer a question. This model answers questions in two distinct modes, and they are not equally reliable:

Open-book (recommended) — you provide a contract excerpt or passage in the prompt along with your question, and the model extracts/classifies/ answers from that given text. This is the mode this revision specifically strengthens, and it is measurably the more trustworthy one (see Evaluation).
Closed-book — you ask a general legal/financial question with no source text, and the model answers from whatever it retained during training. This mode is measurably weaker and prone to confident fabrication (see below) — it is a known, structural limitation of a 125M model recalling specific, often single-occurrence facts from a 2B-token pretraining corpus, not something SFT data changes can fix.

Model description

Same architecture and parameter count as the base model — full fine-tuning does not change the model's shape, only its weights:

Table

Parameters	125,848,320 (~125.8M, tied embeddings)
Architecture	Llama-style decoder, 12 layers / 768 hidden / 12 heads (head dim 64, full MHA)
MLP	SwiGLU, intermediate size 3,072
Positional encoding	RoPE, θ = 10,000
Vocabulary	16,384 byte-level BPE (same tokenizer as the base model, plus role tokens)
Context length	1,024 tokens
Precision	bf16 compute · fp32 master weights and saved checkpoint
Base checkpoint	`DeependraVerma/slm-125m-base` (this author's own from-scratch pretrain, not a third-party checkpoint)

Intended uses & limitations

Known limitations — read before using

A full held-out evaluation (all validation examples, not a spot-check — see Evaluation below) plus manual inspection of failure cases found:

Open-book contract tasks work reasonably well. Given a contract excerpt, the model extracts the relevant clause (or correctly says a clause type isn't present) most of the time, and classifies clause categories correctly roughly 7 times out of 10.
Closed-book general Q&A (no source text given) is weak and prone to confident, specific fabrication. Manual review of failed held-out examples found the model inventing plausible-but-wrong specifics — e.g., asked what a doctor's diagnosis was in a case excerpt, it confidently stated a specific but entirely fabricated clinical detail that contradicted the real answer; asked about a case holding, it invented a different, incorrect legal conclusion. This is not evasive or hedged — it states invented specifics with the same confidence as correct ones, which is the actual risk with a model this size doing closed-book fact recall.
This is a capacity limitation, not a data quality issue. A controlled test (retraining on the original, smaller closed-book-only dataset and re-measuring with the same methodology) showed the same weakness was already present before this revision's data additions — it reflects what a 2-epoch, 2B-token pretrain at this parameter count can and cannot retain about specific, often single-occurrence facts, not a defect introduced by any one training run.

Training data

Two complementary sources of supervision:

Open-book contract pairs (new this revision) — two public, CC BY 4.0 datasets, used as ground truth directly (no LLM teacher, no synthetic generation):

theatticusproject/cuad-qa — expert-annotated commercial contract clause extraction, 41 clause categories across hundreds of real EDGAR-filed contracts. Answer excerpts are windowed around the true answer span (not truncated from the top of the document) so every training example is actually grounded in what the model is shown.
coastalcph/lex_glue (ledgar config) — labeled contract provisions, 100 legal clause categories, SEC contract-exhibit sourced.

Table

Final curated pairs	21,186 (20,127 train / 1,059 val)
Source mix	CUAD (contracts) ~51% · LEDGAR (contracts) ~22% · SEC filings ~13% · US case law ~12% · educational web ~2.5%
Task mix	extraction ~60% · classification ~22% · QA ~14% · summarization ~5% · rewrite <1%
Dedup / decontamination	exact + MinHash-LSH near-dup removal, scoped per source document so identical question templates across different contracts aren't collapsed together; disjoint train/val split by construction

Training procedure

Table

Method	full fine-tune (not LoRA), from the base checkpoint
Hardware	1 GPU (this project's on-prem 8×B200 box)
Epochs	2
Learning rate	3e-5, cosine decay, 3% warmup
Precision	bf16 compute, fp32 master weights
Loss masking	answer tokens only — prompt (system + user turns, including any contract excerpt) masked to `-100`
Wall-clock time	~166 seconds

Chat format — same as the base project's convention, with an optional excerpt folded into the user turn for open-book examples:

markdown
<|bos|><|system|>{system}<|user|>{excerpt (if any)}\n\nQuestion: {question}<|assistant|>{answer}<|eos|>

Default system prompt: "You are a knowledgeable legal and financial assistant. Answer accurately and concisely."

Evaluation

Table with columns: Task, Metric, Result
Task	Metric	Result
CUAD contract clause extraction (open-book)	Token-F1	0.711
CUAD "clause not present" refusal (open-book)	Accuracy	90.1%
LEDGAR clause classification (open-book), 100 categories	Exact-match accuracy	74.1%
Case law general Q&A (mostly open-book after this revision)	Llama-70B-judged correctness	35.6%
SEC filings general Q&A (mostly open-book after this revision)	Llama-70B-judged correctness	43.1%

How to use

Open-book (recommended) — provide the excerpt in the prompt:

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("DeependraVerma/legal-slm-125m-sft")
model = AutoModelForCausalLM.from_pretrained(
    "DeependraVerma/legal-slm-125m-sft", torch_dtype=torch.bfloat16
)

system = "You are a knowledgeable legal and financial assistant. Answer accurately and concisely."
excerpt = "This Agreement may be terminated by either party upon 30 days written notice..."
question = f"{excerpt}\n\nQuestion: What are the termination terms of this agreement?"

def sid(t):
    return tok.convert_tokens_to_ids(t)

ids = (
    tok("<|bos|>", add_special_tokens=False)["input_ids"]
    + [sid("<|system|>")] + tok(system, add_special_tokens=False)["input_ids"]
    + [sid("<|user|>")] + tok(question, add_special_tokens=False)["input_ids"]
    + [sid("<|assistant|>")]
)
out = model.generate(
    torch.tensor([ids]), max_new_tokens=120, do_sample=True,
    temperature=0.7, top_p=0.9, eos_token_id=sid("<|eos|>"), pad_token_id=sid("<|pad|>"),
)
print(tok.decode(out[0][len(ids):], skip_special_tokens=True))

For browser/edge inference (transformers.js, no server), see DeependraVerma/legal-slm-125m-sft-onnx — note that export predates this revision and has not yet been regenerated from these weights.

Citation

bibtex
@misc{verma2026legalslm125msft,
  author = {Deependra Verma},
  title  = {legal-slm-125m-sft: A Fine-Tuned Q\&A Assistant on a From-Scratch 125M Legal/Financial Language Model},
  year   = {2026},
  url    = {https://huggingface.co/DeependraVerma/legal-slm-125m-sft},
  note   = {Code: https://github.com/DeependraVerma/legal-slm-125M}
}

Author

Deependra Verma — Generative AI Researcher / AI Engineer. GitHub · Hugging Face

legal-slm-125m-sft

README

What this is

Model description

Intended uses & limitations

Known limitations — read before using

Training data

Training procedure

Evaluation

How to use

Citation

Author

Explore FriendliAI today

README

What this is

Model description

Intended uses & limitations

Known limitations — read before using

Training data

Training procedure

Evaluation

How to use

Citation

Author