GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
Kotichitturu
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
Kotichitturu
Model Tree
Input Modalities
Output Modalities
Supported Functionality
python
from transformers import AutoModelForCausalLM, AutoTokenizertok = AutoTokenizer.from_pretrained("Kotichitturu/slm-125m-base")model = AutoModelForCausalLM.from_pretrained("Kotichitturu/slm-125m-base")prompt = "The plaintiff alleges that the defendant breached" # a PREFIX, not a questionids = tok(prompt, return_tensors="pt")out = model.generate(**ids, max_new_tokens=90,do_sample=True, temperature=0.8, top_p=0.95, top_k=50, # sampling matters — see below)print(tok.decode(out[0], skip_special_tokens=True))
Use sampling, not greedy. With do_sample=False this model degenerates into
loops ("…minimum net worth????????"). temperature≈0.8, top_p≈0.95 is a sane
default. That is normal for a small base model, not a bug in these weights.
<|system|>, <|user|>, <|assistant|> exist in the vocabulary — they were added
when the tokenizer was trained, so that fine-tuning later would not need a vocab
resize (which would invalidate every embedding). But the pretraining corpus is raw
text and never contains those strings, so their embeddings received essentially no
gradient. They are untrained vectors in this model. Sending a chat-formatted prompt
here produces noise. Use the SFT models above for chat format.
| Architecture | LLaMA (decoder-only, pre-norm) |
| Parameters | 125,848,320 (~125.8M), tied embeddings |
| Layers / hidden / heads | 12 / 768 / 12 (head dim 64, MHA — not GQA) |
| MLP | SwiGLU, inner 3,072 |
| Positions | RoPE (theta 10,000) |
| Norm | RMSNorm, pre-norm |
| Context | 1,024 tokens |
| Vocab | 16,384 byte-level BPE, trained on this corpus |
| Precision | bf16 |
| Data seen | 2.04B unique tokens × 3 epochs = 6.12B |
| Optimizer | AdamW, lr 6e-4 → 6e-5 cosine, 200M-token warmup |
| Batch | 524,288 tokens global |
| Hardware | 8×H100 (Modal), ~1.96M tok/s |
| Steps | 11,667 |
| Wall time | ~52 minutes |
| Final train loss | 2.1899 |
| Final val loss | 2.2521 |
| Perplexity | 9.51 |
Perplexity 9.51 means that, on held-out text from this domain, the model is on average choosing between about 9.5 equally likely next tokens. Reasonable for 125M parameters on specialised text.
Re-measured later on an independent harness at 9.50 (loss 2.2511) — the 0.001 difference is bf16 rounding. That number is the control for the alignment-tax comparison of the fine-tuned children: Q&A costs +1.6% perplexity, RAFT +6.0%.
| Source | Share | Dataset |
|---|---|---|
| US case law | ~40% | HFforLegal/case-law |
| SEC filings (10-K, 10-Q) | ~40% | PleIAs/SEC |
| Educational web | ~20% | HuggingFaceFW/fineweb-edu (sample-10BT) |
Cleaned (language ID, OCR gate, repetition filters), deduplicated (exact + MinHash near-dup), and decontaminated against CaseHOLD and LexGLUE by 13-gram overlap, so downstream evaluation on those benchmarks is not compromised.
Full write-up — data pipeline, tokenizer, packing, the 8×H100 run, deployment, and the fine-tuning that followed — is in the project's study guide. Total spend for the base: **34.80∗∗(3.55 data pipeline + $31.25 pretraining).