GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
Bias-variance-tradeoff
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
Bias-variance-tradeoff
Model Tree
Input Modalities
Output Modalities
Supported Functionality
Prompt it with the opening of a sentence and it completes the thought in a legal or financial style. It is a demonstration of the full from-scratch pipeline (data → tokenizer → pretraining), not a production model.
Not intended for: factual question answering, chat, instruction following, or any high-stakes legal/financial decision-making. At 125M parameters it holds very little factual knowledge and will confidently produce plausible-sounding but incorrect content.
Maps 1:1 to transformers.LlamaConfig.
| Component | Value |
|---|---|
| Architecture | Llama-style decoder-only transformer |
| Layers | 12 |
| Hidden size | 768 |
| Attention heads | 12 (head dim 64), MHA (kv-heads = 12) |
| MLP | SwiGLU, inner dim 3072 |
| Normalization | RMSNorm (pre-norm), eps 1e-5 |
| Positional encoding | RoPE (theta 10000) |
| Context length | 1024 tokens |
| Vocabulary | 16,384 (byte-level BPE), tied embeddings |
| Attention bias | none |
Streamed from HuggingFace, cleaned, deduplicated, and decontaminated. Realized mix (by real tokens):
| Source | Content | Share |
|---|---|---|
| HFforLegal/case-law | US court opinions | ~33% |
| PleIAs/SEC | SEC filings (10-K, etc.) | ~39% |
| HuggingFaceFW/fineweb-edu | Educational web text | ~28% |
Pipeline: stream → deterministic rule-based cleaning (line filters, boilerplate strip, repetition/language/OCR gates) → exact + MinHash near-dedup → 13-gram decontamination against CaseHOLD / LexGLUE eval sets → 16K byte-level BPE tokenizer → pack into 1024-token windows (99/1 train/val split). Final corpus: 2.18B train + 22.0M val tokens.
| Hyperparameter | Value |
|---|---|
| Objective | Next-token cross-entropy (causal LM) |
| Epochs | 1 (2.18B tokens seen) |
| Optimizer | AdamW, betas (0.9, 0.95), weight decay 0.1 |
| Learning rate | 6e-4 → 6e-5, cosine decay |
| Warmup | 200M tokens |
| Gradient clip | 1.0 |
| Global batch | 524,288 tokens/step (micro-batch 32 × seq 1024, grad-accum 2) |
| Precision | bf16 |
| Hardware |
The reference workshop trains ~5 epochs (→ val perplexity 8.50). This checkpoint is a single-epoch run (val perplexity 10.87); more epochs over the same fixed corpus lower perplexity further.
Held-out validation perplexity is exp(mean token-level cross-entropy) over the 1%
val split (21,505 windows, ~22.0M tokens):
Validation perplexity: 10.87 (val loss 2.3856).
The model uses custom special tokens; prepending <|bos|> matches how it was served and
gives the best continuations.
python
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizerrepo = "Bias-variance-tradeoff/slm-125m-base"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.float32).eval()prompt = "The plaintiff alleges that the defendant"bos = tok.convert_tokens_to_ids("<|bos|>")eos = tok.convert_tokens_to_ids("<|eos|>")ids = torch.tensor([[bos] + tok.encode(prompt, add_special_tokens=False)])out = model.generate(ids, max_new_tokens=90, min_new_tokens=40, do_sample=True,temperature=0.8, top_k=50, top_p=0.95, repetition_penalty=1.3,eos_token_id=eos, pad_token_id=eos,)print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
Trained from scratch as part of the Vizuara AI Labs "SLM from scratch" workshop, following the pipeline in Vizuara-AI-Lab/slm-125m-from-scratch.
| 8×H100 (single-node DDP), ~24 min |