srinivasch87

slm125live-base

Model Details

Architecture: Llama-style decoder-only transformer (RoPE, RMSNorm, SwiGLU MLP, tied embeddings)
Parameters: ~125.8M
Hidden size: 768
Layers: 12
Attention heads: 12 (head dim 64, standard multi-head attention — no GQA)
Intermediate (MLP) size: 3,072
Context length: 1,024 tokens
Vocabulary: 16,384 tokens (custom-trained tokenizer)
RoPE theta: 10,000
Precision: trained and served in fp32/bf16 on Modal

Training Data

Pretrained on a cleaned, deduplicated, decontaminated corpus of ~2.19B tokens, mixed "legal-first" from three public, streamed HuggingFace datasets:

Table with columns: Source, Dataset, Share, What it is
Source	Dataset	Share	What it is
case-law	`HFforLegal/case-law` (`us` config)	~39% (~863M tokens)	US court opinions (scanned; some OCR noise)
sec	`PleIAs/SEC`	~39% (~861M tokens)	SEC filings (10-K, etc.), born-digital
fineweb-edu	`HuggingFaceFW/fineweb-edu` (`sample-10BT`)

The two legal sources were taken in full (they cap out around 2B tokens combined); a smaller web slice was added on top, landing at roughly a 40/40/20 split — about 78% legal/financial text overall, not the originally-planned 70/20/10.

How to Get Started

python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "srinivasch87/slm125live-base"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

prompt = "The plaintiff argued that"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(
    **inputs,
    max_new_tokens=128,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
    top_k=50,
)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Intended Use & Limitations

This is a base completion model, not an instruction-following assistant. It will not reliably answer questions, follow chat-style instructions, or refuse unsafe requests — prompt it with the start of a sentence or document.
At 125M parameters and ~2.2B training tokens, factual accuracy, coherence over long spans, and reasoning are limited. Treat all output as unverified autocomplete, not legal, financial, or professional advice.
Training data skews heavily toward US case law and SEC filings, so the model's style and any biases reflect that domain (and the OCR noise present in the scanned case-law source).
License is not specified for this model; the underlying training datasets each carry their own terms — check the dataset cards linked above before redistributing outputs.

Training & Serving Infrastructure

Built end-to-end on Modal: data cleaning, deduplication, tokenizer training, and pretraining all ran as fanned-out CPU/GPU Modal functions against a shared Modal Volume. Inference is served from a Modal web endpoint with token-streaming generation, behind a Next.js frontend deployed on Vercel (link above).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

srinivasch87

Model Tree

Base

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

Model Details

Architecture: Llama-style decoder-only transformer (RoPE, RMSNorm, SwiGLU MLP, tied embeddings)
Parameters: ~125.8M
Hidden size: 768
Layers: 12
Attention heads: 12 (head dim 64, standard multi-head attention — no GQA)
Intermediate (MLP) size: 3,072
Context length: 1,024 tokens
Vocabulary: 16,384 tokens (custom-trained tokenizer)
RoPE theta: 10,000
Precision: trained and served in fp32/bf16 on Modal

Training Data

Pretrained on a cleaned, deduplicated, decontaminated corpus of ~2.19B tokens, mixed "legal-first" from three public, streamed HuggingFace datasets:

Table with columns: Source, Dataset, Share, What it is
Source	Dataset	Share	What it is
case-law	`HFforLegal/case-law` (`us` config)	~39% (~863M tokens)	US court opinions (scanned; some OCR noise)
sec	`PleIAs/SEC`	~39% (~861M tokens)	SEC filings (10-K, etc.), born-digital
fineweb-edu	`HuggingFaceFW/fineweb-edu` (`sample-10BT`)

How to Get Started

python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "srinivasch87/slm125live-base"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

prompt = "The plaintiff argued that"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(
    **inputs,
    max_new_tokens=128,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
    top_k=50,
)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Intended Use & Limitations

This is a base completion model, not an instruction-following assistant. It will not reliably answer questions, follow chat-style instructions, or refuse unsafe requests — prompt it with the start of a sentence or document.
At 125M parameters and ~2.2B training tokens, factual accuracy, coherence over long spans, and reasoning are limited. Treat all output as unverified autocomplete, not legal, financial, or professional advice.
Training data skews heavily toward US case law and SEC filings, so the model's style and any biases reflect that domain (and the OCR noise present in the scanned case-law source).
License is not specified for this model; the underlying training datasets each carry their own terms — check the dataset cards linked above before redistributing outputs.

slm125live-base

README

Model Details

Training Data

How to Get Started

Intended Use & Limitations

Training & Serving Infrastructure

Explore FriendliAI today

README

Model Details

Training Data

How to Get Started

Intended Use & Limitations

Training & Serving Infrastructure