Ace-2504

slm-500m-sft

README

License: apache-2.0

At a glance

Table

Base model	`thesreedath/slm-500m-base`
Method	full fine-tune
Parameters	517.8M
Trainable parameters	517.8M (all)
Training data	15,000 grounded QA pairs
Schedule	3 epochs · 2,812
Compute	31.6 min on L4 (Modal)
Cost	$1.03 total lineage

Usage

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "Ace-2504/slm-500m-sft"
model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval()
tok = AutoTokenizer.from_pretrained(MODEL)

SYS = ('You are a precise legal and financial assistant. Answer clearly using the '
       'provided context; do not invent facts.')
user = 'Context:\n<passage>\n\nQuestion: <your question>'

# Custom chat scheme. NOTE: never prepend <|bos|> — it was left untrained.
prompt = f'<|system|>\n{SYS}<|eos|>\n<|user|>\n{user}<|eos|>\n<|assistant|>\n'
enc = tok(prompt, return_tensors='pt', add_special_tokens=False).to(model.device)
eos = tok.convert_tokens_to_ids('<|eos|>')
out = model.generate(**enc, max_new_tokens=160, eos_token_id=eos,
                     pad_token_id=eos)   # pad and eos share an id here
print(tok.decode(out[0, enc['input_ids'].shape[1]:], skip_special_tokens=True))

Evaluation

This checkpoint was not evaluated. The published study evaluated the base, DPO and RLAIF legs of the 500M line; this intermediate checkpoint was trained and released for completeness but never scored, so no numbers are quoted here rather than borrowed from a sibling model.

Architecture

Table

Class	LlamaForCausalLM
Layers	24
Hidden size	1,280
Attention	20 heads · head dim 64 · full MHA
Feed-forward	SwiGLU · inner 3,456
Positional	RoPE · θ 10,000
Norm	RMSNorm · ε 1e-5
Context	1,024 tokens
Vocabulary	32,768

Training

Initialised from thesreedath/slm-500m-base
Method — full fine-tune
Data — 15,000 grounded QA pairs, generated from a legal/financial corpus and gated by an LLM judge for faithfulness
Schedule — 3 epochs · 2,812, 31.6 min on L4
Loss is masked to the answer tokens only, so the model is never trained to reproduce the prompt

**Cost — $1.03. * * T h e ba se w a s im p or t e d, soo n l y t hi s p r o j ec t' so w n w or k i sco u n t e d :$ 1.03 of Modal GPU time and shared dataset generation.

Limitations and intended use

Not legal or financial advice. This is a research artefact for studying small-model training, not a professional tool. Do not rely on its output for real decisions.
At 517.8M it holds very little world knowledge. It is built to read an answer out of a passage you supply, not to recall facts. Used closed-book it will produce fluent, confident and wrong text.
The judge behind these scores has not been calibrated against human labels, so treat small differences between models cautiously.
English only; the corpus is US case law, SEC filings and educational web text.

The rest of the family

Table with columns: Size, Base, QA SFT, RAFT, DPO, RLAIF
Size	Base	QA SFT	RAFT	DPO	RLAIF
125M	`slm-125m-e4`	`slm-125m-sft`	`slm-125m-raft`	`slm-125m-sft-dpo`	`slm-125m-sft-rlaif`
500M

All under Ace-2504 except the two imported bases.

Citation

bibtex
@misc{sandhu2026slm,
  title  = {Small Language Models for Legal and Financial Text: a controlled study of
            pretraining, instruction tuning, retrieval augmentation and alignment},
  author = {Harman Sandhu},
  year   = {2026},
  note   = {https://slm-arena-harman.vercel.app}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Ace-2504

Model Tree

Base

thesreedath/slm-500m-base

Fine-tuned

this model

Input Modalities

Text

Output Modalities