genaiquest

slm-125m-base

README

License: apache-2.0

Quickstart

python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("genaiquest/slm-125m-base")
model = AutoModelForCausalLM.from_pretrained("genaiquest/slm-125m-base")

ids = tok("The plaintiff alleges that the defendant", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=60, do_sample=True,
      temperature=0.8, min_new_tokens=40, repetition_penalty=1.3)[0]))

Suggested sampling: temperature 0.8 · top_p 0.95 · repetition_penalty 1.3 (small base models loop without a repetition penalty).

Prompts that work well (in-register continuations):

IN THE UNITED STATES DISTRICT COURT
Pursuant to Section 10(b) of the Securities
The Company's revenue for the fiscal year
The court finds that the evidence presented

Architecture

Deliberately vanilla geometry, mapping 1:1 to transformers.LlamaConfig — so it exports to ONNX (and GGUF) with zero custom-converter work. That design choice is exactly what makes the in-browser playground possible.

Table with columns: Component, Spec
Component	Spec
Family	Llama-style decoder-only
Hidden size	768
Layers	12
Attention heads	12 (head dim 64)
KV heads	12 — MHA, `num_kv == num_heads` → export-safe
MLP	SwiGLU, intermediate 3072 (4×), SiLU gate
Normalization	RMSNorm (eps 1e-5), pre-norm
Positional encoding	RoPE, θ = 10,000

Parameter budget (tied):

markdown
embed      16,384 × 768        = 12.58M
attn/layer 4 × 768²            =  2.36M
mlp/layer  3 × 768 × 3072      =  7.08M
(attn + mlp) × 12 layers       = 113.2M
total                          ≈ 125.8M

Training

Data: US case law + SEC filings + a general web slice, deduplicated; ~2.19B tokens total. Case law and SEC filings are public-domain US government works. [40%-40%-20%]
Tokenizer: 16,384-token BPE trained from scratch on the corpus.
Compute: 1 epoch on 8×H100. [FILL: wall-clock hours, tokens/sec, approx cost — e.g., "X hrs ≈ $Y at market spot rates"]
Recipe: optimizer (AdamW), peak LR, schedule (cosine), 200M token warmup, global batch size 32, 0.5M tokens/step, grad clip, weight decay
Scaling note: 2.19B tokens ÷ 125.8M params ≈ 17.5 tokens/param, close to the Chinchilla compute-optimal ratio (~20) — this run prioritized compute-optimality over deliberate overtraining.

Loss curve

Evaluation

Table with columns: Metric, Value
Metric	Value
Validation perplexity (held-out, in-domain)	11.01

Perplexity is measured on a held-out slice of the training distribution (legal/financial register). Expect materially higher perplexity on out-of-domain text; this model is a domain specialist by construction.

Intended use & limitations

Intended: studying small-model pretraining end to end; domain-register text continuation; a base checkpoint for SFT/DPO/RAG experiments; edge/browser inference demos; teaching material for the full train→quantize→deploy pipeline.

Not intended: legal or financial advice, factual question answering, chat, or any production decision-making. Outputs can be fluent and wrong — at 125M parameters that is a certainty, not a risk. Generated text may echo the style of court filings and SEC documents; verify nothing here as fact.

ONNX / browser deployment

The repo includes an int8 (q8) ONNX export consumed by 🤗 Transformers.js in the playground. Because the architecture maps 1:1 to LlamaConfig, the export pipeline is: bf16 safetensors → ONNX → dynamic int8 quantization, with no custom conversion code.

Roadmap

This is checkpoint 1 of a series building the full enterprise-SLM stack in public:

✅ slm-125m-base — pretraining from scratch (this model)
⏳ -sft — supervised fine-tuning for instruction following / QnA
⏳ -dpo/ppo — preference tuning
⏳ RAG-grounded variants for document QnA with citations
⏳ Serving optimization study on this model family (vLLM, CUDA graphs, FP8) — companion to my vLLM H100 serving experiments

Author

Bhabani Nayak — AI engineering leader; agentic AI at consumer scale (Sephora; previously Shopify, Twitter, Apple). IEEE Senior Member. LinkedIn · Substack · Playground

Citation

bibtex
@misc{nayak2026slm125m,
  author = {Nayak, Bhabani},
  title  = {slm-125m-base: a 125M Llama-style language model pretrained from scratch on legal/financial text},
  year   = {2026},
  url    = {https://huggingface.co/genaiquest/slm-125m-base}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

genaiquest

Model Tree

Base

this model

Input Modalities

Text

Output Modalities