DeependraVerma

legal-slm-500m-sft

What this is

Same two-mode design as the 125M SFT model:

Open-book (recommended) — give it a contract excerpt or passage along with your question; it extracts/classifies/answers from that given text. This is the mode with real, measured reliability.
Closed-book — a general legal/financial question with no source text. Meaningfully better than the 125M model here too, but still the weaker, riskier mode — see Known limitations.

Model description

Table

Parameters	528,538,752 (~528.5M, tied embeddings)
Architecture	Llama-style decoder, 24 layers / 1152 hidden / 18 heads (head dim 64, full MHA)
MLP	SwiGLU, intermediate size 4,608
Vocabulary	16,384 byte-level BPE (same tokenizer as the 125M build, plus role tokens)
Context length	1,024 tokens
Base checkpoint	`DeependraVerma/slm-500m-base` (this author's own from-scratch pretrain)

Training data

Deliberately reuses the exact same curated SFT dataset as the 125M model — 21,186 pairs: CUAD contract clause extraction (CC BY 4.0), LEDGAR clause classification (CC BY 4.0), and open-book QA/summarization/extraction over case-law/SEC/educational-web passages, distilled via a local Meta-Llama-3.1-70B-Instruct teacher. Reusing the same dataset (rather than rebuilding it) is valid because the tokenized data depends only on the tokenizer, which both models share — it isolates the comparison to "does a bigger, better-pretrained base model produce a better SFT result," not "is the SFT data different."

Table

Train / val split	20,127 / 1,059
Method	full fine-tune (not LoRA), 2 epochs
Epoch 1 val_loss → epoch 2 val_loss	0.1840 → 0.1887 (ticked up — a real, measured overfitting signal on this dataset size; do not push epochs further without new data)

Evaluation

Full held-out validation set (1,059 examples), same methodology as the 125M model's card — extraction/classification against ground truth, general QA judged by an independent local Meta-Llama-3.1-70B-Instruct:

Table with columns: Task, 125M, 500M
Task	125M	500M
CUAD contract clause extraction (Token-F1)	0.711	0.761
CUAD "clause not present" refusal accuracy	90.1%	87.7%
LEDGAR clause classification (exact-match)	74.1%	77.4%
Case law general Q&A (closed-book)	35.6%	42.2%
SEC filings general Q&A (closed-book)	43.1%	46.9%
Educational web general Q&A (closed-book)

A direct, paired, per-question comparison (both models answering the identical 1,059 questions) found the 500M model wins net in every single category, but not as a clean sweep — genuine regressions exist alongside the improvements (e.g. CUAD: 46 questions fixed, 25 newly wrong that the 125M model had gotten right). One concrete regression, worth citing directly: asked what changed conditions a defendant cited, the 125M model correctly listed the real conditions from the source text; the 500M model produced a vaguer, legal-sounding but non-responsive answer. Bigger and better on net does not mean strictly better on every question.

Known limitations — read before using

Open-book contract tasks work well and are the trustworthy mode — measurably better than the 125M model across the board.
Closed-book general Q&A is improved but still fundamentally limited. A 528.5M-parameter model can store at most ~2 bits of knowledge per parameter (Allen-Zhu & Li, "Physics of Language Models: Knowledge Capacity Scaling Laws") — roughly 132MB of total compressible fact storage, shared across everything it knows. Combined with the fact that most specific facts in its training data (a case's dollar figure, a specific statute citation) appeared only once or twice — and memorization research (Carlini et al., "Quantifying Memorization Across Neural Language Models") shows once-seen facts are memorized only ~0.75% of the time versus 40%+ for facts repeated 500+ times — closed-book precision on rare facts is a hard capacity limitation, not something more training on this same recipe fixes.
Confident fabrication is real and was directly observed, not hypothetical: specific invented clinical/legal details contradicting the real source, and the vague-non-answer regression cited above.

Never use this model's output as legal, financial, or factual advice. Always treat specific claims as unverified until checked against a primary source, especially in closed-book use.

External benchmark evaluation

This project's own eval scripts (above) are one perspective. To check the same claims against outside measurement, both this model and its 125M predecessor were run through lm-evaluation-harness (the same framework behind a widely used public LLM leaderboard) on five benchmark categories:

Table with columns: Benchmark, Result, Baseline, Read
Benchmark	Result	Baseline	Read
HellaSwag / ARC-Easy / PIQA (general commonsense)	32.4% / 44.5% / 59.6% (acc_norm)	n/a	Normal range for a model this size — not a legal claim, a peer-comparison reference point
MMLU professional_law / jurisprudence / international_law	24.7% / 25.9% / 24.0% (acc)	25% (4-choice)	At the random-chance floor — no real legal knowledge memorized, exactly as the caveats above already say
CaseHOLD (pick the correct legal holding, via LexGLUE)	19.9% (acc_norm)	20% (5-choice)	Right at random chance — real closed-book legal reasoning is not something this model can do, stated as plainly as a number can
LegalBench — 12-task contract NLI/QA subset

LegalBench isn't in lm-eval-harness's default task set, so a 12-task, contract-focused subset was hand-configured using the benchmark's own official prompts, and checked for training-data contamination with an n-gram overlap scan against the full pretraining corpus before trusting the result (a naive check first over-flagged common legal boilerplate — standard-of-review language, ToS disclaimer text — as "contamination"; a stricter re-check traced the real overlap to 5 of 1,853 documents, all explainable — e.g. a real, publicly available company's Terms of Service — and none affecting the reported numbers). SCOTUS, ECtHR, and EUR-LEX (also part of LexGLUE) were excluded: those documents run tens of thousands of tokens, far beyond this model's 1,024-token context, and scoring on a truncated sliver of a document isn't a meaningful number.

Full methodology, task configs, and raw logs: github.com/DeependraVerma/legal-slm-125M/tree/main/benchmarks.

How to use

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("DeependraVerma/legal-slm-500m-sft")
model = AutoModelForCausalLM.from_pretrained(
    "DeependraVerma/legal-slm-500m-sft", torch_dtype=torch.bfloat16
)

system = "You are a knowledgeable legal and financial assistant. Answer accurately and concisely."
excerpt = "This Agreement may be terminated by either party upon 30 days written notice..."
question = f"{excerpt}\n\nQuestion: What are the termination terms of this agreement?"

def sid(t):
    return tok.convert_tokens_to_ids(t)

ids = (
    tok("<|bos|>", add_special_tokens=False)["input_ids"]
    + [sid("<|system|>")] + tok(system, add_special_tokens=False)["input_ids"]
    + [sid("<|user|>")] + tok(question, add_special_tokens=False)["input_ids"]
    + [sid("<|assistant|>")]
)
out = model.generate(
    torch.tensor([ids]), max_new_tokens=120, do_sample=True,
    temperature=0.7, top_p=0.9, eos_token_id=sid("<|eos|>"), pad_token_id=sid("<|pad|>"),
)
print(tok.decode(out[0][len(ids):], skip_special_tokens=True))

Citation

bibtex
@misc{verma2026legalslm500msft,
  author = {Deependra Verma},
  title  = {legal-slm-500m-sft: A Fine-Tuned Q\&A Assistant on a From-Scratch 528.5M Legal/Financial Language Model},
  year   = {2026},
  url    = {https://huggingface.co/DeependraVerma/legal-slm-500m-sft},
  note   = {Code: https://github.com/DeependraVerma/legal-slm-125M}
}

Author

Deependra Verma — Generative AI Researcher / AI Engineer. GitHub · Hugging Face

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

DeependraVerma

Model Tree

Base

DeependraVerma/slm-500m-base

Fine-tuned

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

What this is

Same two-mode design as the 125M SFT model:

Open-book (recommended) — give it a contract excerpt or passage along with your question; it extracts/classifies/answers from that given text. This is the mode with real, measured reliability.
Closed-book — a general legal/financial question with no source text. Meaningfully better than the 125M model here too, but still the weaker, riskier mode — see Known limitations.

Model description

Table

Parameters	528,538,752 (~528.5M, tied embeddings)
Architecture	Llama-style decoder, 24 layers / 1152 hidden / 18 heads (head dim 64, full MHA)
MLP	SwiGLU, intermediate size 4,608
Vocabulary	16,384 byte-level BPE (same tokenizer as the 125M build, plus role tokens)
Context length	1,024 tokens
Base checkpoint	`DeependraVerma/slm-500m-base` (this author's own from-scratch pretrain)

Training data

Table

Train / val split	20,127 / 1,059
Method	full fine-tune (not LoRA), 2 epochs
Epoch 1 val_loss → epoch 2 val_loss	0.1840 → 0.1887 (ticked up — a real, measured overfitting signal on this dataset size; do not push epochs further without new data)

Evaluation

Table with columns: Task, 125M, 500M
Task	125M	500M
CUAD contract clause extraction (Token-F1)	0.711	0.761
CUAD "clause not present" refusal accuracy	90.1%	87.7%
LEDGAR clause classification (exact-match)	74.1%	77.4%
Case law general Q&A (closed-book)	35.6%	42.2%
SEC filings general Q&A (closed-book)	43.1%	46.9%
Educational web general Q&A (closed-book)

Known limitations — read before using

Open-book contract tasks work well and are the trustworthy mode — measurably better than the 125M model across the board.
Closed-book general Q&A is improved but still fundamentally limited. A 528.5M-parameter model can store at most ~2 bits of knowledge per parameter (Allen-Zhu & Li, "Physics of Language Models: Knowledge Capacity Scaling Laws") — roughly 132MB of total compressible fact storage, shared across everything it knows. Combined with the fact that most specific facts in its training data (a case's dollar figure, a specific statute citation) appeared only once or twice — and memorization research (Carlini et al., "Quantifying Memorization Across Neural Language Models") shows once-seen facts are memorized only ~0.75% of the time versus 40%+ for facts repeated 500+ times — closed-book precision on rare facts is a hard capacity limitation, not something more training on this same recipe fixes.
Confident fabrication is real and was directly observed, not hypothetical: specific invented clinical/legal details contradicting the real source, and the vague-non-answer regression cited above.

Never use this model's output as legal, financial, or factual advice. Always treat specific claims as unverified until checked against a primary source, especially in closed-book use.

External benchmark evaluation

Table with columns: Benchmark, Result, Baseline, Read
Benchmark	Result	Baseline	Read
HellaSwag / ARC-Easy / PIQA (general commonsense)	32.4% / 44.5% / 59.6% (acc_norm)	n/a	Normal range for a model this size — not a legal claim, a peer-comparison reference point
MMLU professional_law / jurisprudence / international_law	24.7% / 25.9% / 24.0% (acc)	25% (4-choice)	At the random-chance floor — no real legal knowledge memorized, exactly as the caveats above already say
CaseHOLD (pick the correct legal holding, via LexGLUE)	19.9% (acc_norm)	20% (5-choice)	Right at random chance — real closed-book legal reasoning is not something this model can do, stated as plainly as a number can
LegalBench — 12-task contract NLI/QA subset

Full methodology, task configs, and raw logs: github.com/DeependraVerma/legal-slm-125M/tree/main/benchmarks.

How to use

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("DeependraVerma/legal-slm-500m-sft")
model = AutoModelForCausalLM.from_pretrained(
    "DeependraVerma/legal-slm-500m-sft", torch_dtype=torch.bfloat16
)

system = "You are a knowledgeable legal and financial assistant. Answer accurately and concisely."
excerpt = "This Agreement may be terminated by either party upon 30 days written notice..."
question = f"{excerpt}\n\nQuestion: What are the termination terms of this agreement?"

def sid(t):
    return tok.convert_tokens_to_ids(t)

ids = (
    tok("<|bos|>", add_special_tokens=False)["input_ids"]
    + [sid("<|system|>")] + tok(system, add_special_tokens=False)["input_ids"]
    + [sid("<|user|>")] + tok(question, add_special_tokens=False)["input_ids"]
    + [sid("<|assistant|>")]
)
out = model.generate(
    torch.tensor([ids]), max_new_tokens=120, do_sample=True,
    temperature=0.7, top_p=0.9, eos_token_id=sid("<|eos|>"), pad_token_id=sid("<|pad|>"),
)
print(tok.decode(out[0][len(ids):], skip_special_tokens=True))

Citation

bibtex
@misc{verma2026legalslm500msft,
  author = {Deependra Verma},
  title  = {legal-slm-500m-sft: A Fine-Tuned Q\&A Assistant on a From-Scratch 528.5M Legal/Financial Language Model},
  year   = {2026},
  url    = {https://huggingface.co/DeependraVerma/legal-slm-500m-sft},
  note   = {Code: https://github.com/DeependraVerma/legal-slm-125M}
}

Author

Deependra Verma — Generative AI Researcher / AI Engineer. GitHub · Hugging Face

legal-slm-500m-sft

README

What this is

Model description

Training data

Evaluation

Known limitations — read before using

External benchmark evaluation

How to use

Citation

Author

Explore FriendliAI today

README

What this is

Model description

Training data

Evaluation

Known limitations — read before using

External benchmark evaluation

How to use

Citation

Author