abhishekai

slm-125m-legal-sft

README

License: apache-2.0

How it was tuned

The base model was fine-tuned on ~9,100 grounded Q&A pairs (plus an 800-pair held-out eval split). The data was generated by a teacher LLM (Gemini 2.5 Flash) reading passages sampled from the same legal/financial corpus the base model was pretrained on, then aggressively filtered:

Grounding / faithfulness — an LLM-judge self-check; only answers scored fully supported by their source passage were kept.
Deduplication — near-identical questions removed by embedding similarity.
Length / format / schema — truncated, empty, or malformed rows dropped.
Task & difficulty balance — five task types (grounded QA, multi-step QA, summarization, extraction, rewriting) at a fixed mix, spanning easy → hard.
Eval decontamination — n-gram and embedding overlap between the training and eval splits removed, so reported eval numbers are not leaked training data.

Reused the base model's own 16K byte-level BPE tokenizer (no re-tokenization). A chat template is bundled and used at both training and inference.

Training procedure

Table

Method	full-parameter SFT, loss on answer tokens only (prompt masked)
Examples	9,096 train / 794 eval
Epochs	3 (1,704 steps, micro-batch 16)
Optimizer	AdamW (beta 0.9/0.95)
LR schedule	3% warmup to 2e-5, cosine decay to 2e-6
Format	chat template; context folded into the user turn, RAFT-style
Hardware	1x H100
Compute cost	~ $0.32 (d a t a g e n er a t i o n$ 2.44 on top)

Evaluation

Cross-entropy over answer tokens only on the held-out eval split:

Table with columns: Metric, Value
Metric	Value
Eval loss (answer tokens)	1.213
Eval perplexity (answer tokens)	3.36

This is not comparable to the base model's 9.155 — that is general next-token perplexity over the whole sequence; this is loss over supervised answer tokens on grounded QA, an easier target. No downstream task benchmarks (LegalBench, CaseHOLD, etc.) have been run.

Usage

Give it a question and a context passage. Greedy decoding is strongly recommended — at higher temperatures a model this small drifts into the wrong task type.

python
from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("abhishekai/slm-125m-legal-sft")
model = AutoModelForCausalLM.from_pretrained("abhishekai/slm-125m-legal-sft").eval()

question = "What standard of proof applies, and what must the plaintiff show?"
context = (
    "In a civil negligence action the plaintiff must establish duty, breach, "
    "causation, and damages. The standard of proof is a preponderance of the "
    "evidence: the plaintiff must show it is more likely than not that the "
    "defendant's conduct caused the harm."
)
messages = [
    {"role": "system", "content": "You are a precise legal and financial assistant. "
        "Answer using only the provided context. If the context does not contain the "
        "answer, say you cannot answer from the context."},
    {"role": "user", "content": f"{question}\n\nContext:\n{context}"},
]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids = tok(text, add_special_tokens=False, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=160, do_sample=False,
                     eos_token_id=tok.convert_tokens_to_ids("<|eos|>"),
                     pad_token_id=tok.convert_tokens_to_ids("<|pad|>"))
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

Limitations and intended use

A research and educational artifact — a demonstration that a coherent, grounded instruction-following model can be distilled onto a 125M base for a few dollars. Not suitable for production, and specifically:

It needs a context passage. It was trained to answer from provided context, not from parametric memory. Closed-book questions are out of distribution.
It does not do arithmetic. It will confidently state figures and percentages that do not follow from each other. Never trust a number it computes.
It still hallucinates and is not a source of legal or financial fact, and is not legal or financial advice. Do not deploy it in any advisory, compliance, or decision-making capacity.
Use greedy / low temperature. Sampling at higher temperatures makes a model this small slip into the wrong task (e.g. emitting JSON for a prose question).
1024-token context, so question + passage together must be short.
English only, US-jurisdiction-skewed, and it inherits the biases of both its pretraining corpus and the teacher model that generated its fine-tuning data.

Data provenance and license

The model weights are released under Apache-2.0, inheriting the base model's license.

The fine-tuning data is synthetic, generated by Google's Gemini 2.5 Flash over passages from the base model's pretraining corpus (US case law — CC-BY-4.0; SEC filings — CC0-1.0; FineWeb-Edu — ODC-By 1.0). Synthetic data carries the teacher provider's usage terms; this artifact is published for research and education. If you intend to use it commercially, review Google's Gemini API terms and get your own counsel to look at it. None of this is legal advice.

Reproducibility

The full fine-tuning pipeline — passage sampling, teacher generation, LLM-judge grounding, deduplication, task balancing, eval decontamination, tokenization, and the SFT run — runs on Modal and is reproducible end-to-end.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

abhishekai

Model Tree

Base

abhishekai/slm-125m-legal-base

Fine-tuned

this model

Input Modalities

Text

Output Modalities