Model Details
Table with columns: Field, Value| Field | Value |
|---|
| Repository | asinha08/slm-125m-base |
| Project / Modal app | slm125mLIVE |
| Model type | Decoder-only causal LM, Llama-compatible |
| Parameters | 125,848,320 |
| Context length | 1,024 tokens |
| Vocabulary size | 16,384 |
| Tokenizer | Byte-level BPE trained from the final corpus |
| Hidden size | 768 |
| Layers | 12 |
| Attention heads | 12 |
| KV heads | 12 |
| Intermediate size | 3,072 |
| Activation | silu / SwiGLU |
| Positional encoding | RoPE, theta 10,000 |
| Norm | RMSNorm, eps 1e-5 |
| Embeddings | Tied input/output embeddings |
| Attention bias | false |
| Saved format | safetensors |
| Weight dtype | bfloat16 |
Special tokens:
Table with columns: Token role, Token| Token role | Token |
|---|
| BOS | <|bos|> |
| EOS | <|eos|> |
| PAD | <|pad|> |
| UNK | <|unk|> |
| Extra | <|user|>, <|assistant|>, <|system|> |
The extra chat-style tokens are present in the tokenizer, but the model was not instruction tuned.
Intended Use
This model is intended for:
- Research on small language model pretraining.
- Legal-domain language modeling experiments.
- Continued pretraining, supervised fine-tuning, or evaluation pipelines.
- Text continuation/completion tests.
This model is not intended for:
- Legal advice or compliance decisions.
- Production legal automation without task-specific evaluation and guardrails.
- Chatbot use without instruction tuning and safety tuning.
- Factual question answering where accuracy is required.
Training Data
The corpus was built from three public Hugging Face datasets with a legal-first mix:
Table with columns: Source, HF dataset, Kept docs, Proxy corpus tokens, Tokenized train tokens, Tokenized val tokens, Tokenized mix| Source | HF dataset | Kept docs | Proxy corpus tokens | Tokenized train tokens | Tokenized val tokens | Tokenized mix |
|---|
| Case law | HFforLegal/case-law | 206,684 | 806,244,775 | 714,858,496 | 7,223,296 | |
Data processing included:
- Basic line/document cleaning.
- OCR/noise filtering for case-law data.
- Exact duplicate removal.
- MinHash near-duplicate removal.
- Decontamination against held-out evaluation sources:
coastalcph/lex_glue and casehold/casehold.
- Byte-level BPE tokenizer training on the cleaned corpus.
- Packed fixed-length token windows with sequence length
1,024.
- Validation routing of every 100th packed window, giving an approximately 99/1 train/validation split.
Deduplication and decontamination summary:
Table with columns: Source, Exact duplicates removed, Near duplicates removed, Contaminated docs removed| Source | Exact duplicates removed | Near duplicates removed | Contaminated docs removed |
|---|
| Case law | 0 | 1,606 | 24,002 |
| SEC filings | 1,989 | 0 | 175 |
| FineWeb-Edu |
Training Procedure
Training ran on Modal using 8x H100 GPUs.
Table with columns: Field, Value| Field | Value |
|---|
| GPUs | 8 x H100 |
| Precision | bfloat16 autocast |
| Optimizer | AdamW |
| Learning rate | 6e-4 |
| Minimum LR | 6e-5 |
| Warmup tokens | 200,000,000 |
| Weight decay |
The model has completed 4 full training epochs over the packed train split.
Evaluation
Perplexity is computed as exp(validation_loss).
The reported validation metrics use 4,096 validation windows, or about 4.19M validation tokens, sampled from the internal packed validation split. The full validation split contains 20,119 windows, or 20,601,856 tokens.
Table with columns: Epoch, Status, Epoch tokens, Total tokens seen, Steps, Validation loss, Validation perplexity| Epoch | Status | Epoch tokens | Total tokens seen | Steps | Validation loss | Validation perplexity |
|---|
| 1 | Completed | 2.039B | 2.039B | 3,889 | 3.3730 | 29.17 |
|
The latest completed evaluation is:
Table with columns: Metric, Value| Metric | Value |
|---|
| Validation loss | 2.7765 |
| Validation perplexity | 16.06 |
| Total tokens seen | 8,155,824,128 |
No downstream task benchmark results are reported yet.
Approximate Training Cost
Known pretraining cost by epoch:
Table with columns: Epoch, H100, CPU, Memory, Total| Epoch | H100 | CPU | Memory | Total |
|---|
| 1 | ~$7.18 | Not recorded | Not recorded | ~$7.18+ |
| 2 | ~$9.60 | Not recorded | Not recorded | ~$9.60+ |
| 3 |
Approximate known pretraining total: ~$39.19+. This excludes any unrecorded dataset preparation, storage, network, or earlier CPU-only costs.
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "asinha08/slm-125m-base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
)
prompt = "<|bos|>The plaintiff alleged that"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
output_ids = model.generate(
**inputs,
max_new_tokens=80,
do_sample=True,
temperature=0.8,
top_p=0.95,
eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output_ids[0], skip_special_tokens=False))
Because this is a base model, prompts should be treated as text-completion prefixes. The tokenizer contains <|user|>, <|assistant|>, and <|system|> tokens, but the model has not been trained to follow chat instructions.
Limitations and Risks
- The model is small and undertrained relative to modern production LLMs.
- The model can hallucinate, fabricate citations, and produce legally incorrect text.
- The model is not safe for legal advice, compliance review, contract review, court filings, or any high-stakes use without extensive validation.
- The model was not instruction tuned, RLHF tuned, preference tuned, or safety aligned.
- The training corpus is mostly legal/financial English text plus a smaller web-education slice; performance outside that distribution may be poor.
- The model may reproduce biases, sensitive patterns, or memorized fragments from public data.
- Evaluation currently reports only internal validation perplexity, not downstream legal-task accuracy.
- Source datasets have their own licenses and use constraints; users should verify dataset licenses before commercial or redistributed use.
Reproducibility Notes
Main artifacts were stored on the Modal volume slm125mLIVE:
Table with columns: Artifact, Path| Artifact | Path |
|---|
| Cleaned data | /data/clean |
| Deduplicated corpus | /data/corpus |
| Tokenizer | /data/tokenizer |
| Tokenized train/val windows | /data/tokens |
| Checkpoints | /data/checkpoints |
| Final base checkpoint | |
Latest training summary:
Table with columns: Field, Value| Field | Value |
|---|
| Status | completed |
| Steps | 3,889 / 3,889 |
| Epoch base tokens | 6,116,868,096 |
| Epoch tokens | 2,038,956,032 |
| Total tokens seen | 8,155,824,128 |
| World size | 8 |
Citation
No formal citation is available. If you use this model, cite the Hugging Face repository: