Sudhanshu1985

slm-125m-base

README

License: apache-2.0

Model details

Table

Parameters	125,847,552 (~125.8M)
Architecture	Llama-style (`LlamaForCausalLM`)
Layers	12
Hidden size	768
Attention heads	12 (head dim 64), MHA (12 KV heads)
MLP	SwiGLU, inner 3072
Normalization	RMSNorm, pre-norm
Positional	RoPE (theta 10000)
Context length	1024
Vocab	16,384 (byte-level BPE, trained from scratch)
Embeddings	tied (input = output)
Precision	bf16

Training data

A ~2.04B-token corpus built from three public, ungated sources, streamed and filtered through a deterministic cleaning + dedup + decontamination pipeline:

Table with columns: Source, Content, Share
Source	Content	Share
`HFforLegal/case-law`	US court opinions	~40%
`PleIAs/SEC`	SEC filings (10-K, etc.)	~40%
`HuggingFaceFW/fineweb-edu`	Educational web text

The corpus was decontaminated against CaseHOLD / LexGLUE (13-gram overlap removed) and deduplicated (exact + MinHash/LSH near-duplicate removal).

Training recipe

Table with columns: Knob, Value
Knob	Value
Objective	next-token cross-entropy
Hardware	8× H100 (single-node DDP)
Epochs	1 (~2.04B tokens seen)
Optimizer	AdamW, betas (0.9, 0.95), weight decay 0.1
LR	6e-4 → 6e-5, cosine decay, 200M-token warmup
Global batch	~524,288 tokens/step
Grad clip	1.0
Steps	3,889

Results

Held-out validation perplexity: 11.01 (val loss 2.42) on a 1% held-out split (~20.6M tokens). Perplexity was still descending at the end of the single epoch (step 2000 → 12.54, step 3000 → 11.27, final → 11.01), so additional epochs would lower it further. Total compute cost to train: ~$11.

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Sudhanshu1985/slm-125m-base")
model = AutoModelForCausalLM.from_pretrained("Sudhanshu1985/slm-125m-base")

prompt = "The plaintiff alleges that the defendant"
inputs = tok(prompt, return_tensors="pt")
out = model.generate(
    **inputs,
    max_new_tokens=80,
    min_new_tokens=40,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
    top_k=40,
    repetition_penalty=1.3,
)
print(tok.decode(out[0], skip_special_tokens=True))

Example completions

"The plaintiff alleges that the defendant" → "breached its contract of employment... In order to prove fraud, a party must show: (1) The existence of a confidential relationship; (2) the absence or violation by one party of any duty owed..."
"Pursuant to the terms of this Agreement," → "the parties have agreed as follows: 1. ...all disputes... shall be subject to the jurisdiction of said Court..."
"The Company's net revenues for the fiscal year" → "ended September 30, 1996 were $305.1 million. The operating loss in fiscal 1995 was primarily attributable to..."

Limitations

Base completer, not a chatbot. It continues text; it does not follow instructions or answer questions.
Does not know facts. At 125M parameters a model holds only a small amount of usable knowledge; grounded facts would require retrieval (RAG).
Domain-biased. It speaks the legal register fluently; non-legal prompts drift toward that register.
Trained on US legal/financial + educational web text; may reflect biases in those sources. Not legal or financial advice.

Provenance

Pretrained from random weights (nothing fine-tuned). Pipeline: stream 3 datasets → rule-based cleaning → dedup + decontaminate → 16K byte-level BPE → pack 1024-token windows → pretrain on 8× H100. Based on the Vizuara AI Labs slm-125m-from-scratch recipe.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Sudhanshu1985

Model Tree

Base

this model

Input Modalities

Text

Output Modalities