GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
DeependraVerma
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
DeependraVerma
Model Tree
Input Modalities
Output Modalities
Supported Functionality
slm-125m-base)Run 1's pretraining data never stripped HTML markup from the SEC sources: the
model spent roughly 19.3% of its 591B-token budget (~114B tokens) learning to
emit <font> and  , and leaked that markup into 77.1%
of legal completions. This run fixes that at the source (strip_markup()
strips markup before tokenization, measured 36.95% → 0.00% markup on real
documents) and warm-starts from run 1's weights rather than training from
scratch, so the existing knowledge is kept while the markup habit is
unlearned.
After warm-starting from run 1 and training on markup-stripped data, a repeated in-training probe (4 fixed legal-drafting prompts, checked for leaked EDGAR/HTML markup in the generated text) most recently measured 0% markup leakage at step 1,617,001, down from run 1's 77.1% baseline on the same class of prompt.
This model's context window is also 4096 tokens (run 1 shipped
with a mismatch between its configured window and its declared
max_position_embeddings; this run keeps the two equal by construction, see
the training code's config for details).
| Parameters | 125,847,552 (~125.8M, tied embeddings) |
| Architecture | Llama-style decoder (maps 1:1 to transformers.LlamaConfig) |
| Layers / hidden / heads | 12 / 768 / 12 (head dim 64, full multi-head attention, no GQA/MQA) |
| MLP | SwiGLU, intermediate size 3072 |
| Positional encoding | RoPE, θ = 10,000 |
| Normalization | RMSNorm, ε = 1e-05 |
| Vocabulary | 16,384, byte-level BPE (same tokenizer as every model in this project) |
| Context length | 4,096 tokens |
| Embeddings | tied input/output |
| Precision | bf16 compute (autocast) · fp32 saved checkpoint |
| Final training step | 2,861,021 / 2,861,022 |
| Final training loss (last logged step) | 1.9260 |
Intended use: text continuation / completion in a legal, financial, and general-English register — drafting-style prose, research and educational purposes (tokenizer design, small-model pretraining, warm-start continuation).
Not intended for: question answering, instruction following, or any use
where factual accuracy matters. This is a base language model — it completes
text plausibly and will fabricate case names, statute citations, dollar
figures, and other specifics that sound right but are not grounded in
anything. For a model that at least attempts to answer questions, see
legal-slm-125m-ultimate-sft.
Same-tokenizer, markup-stripped legal/financial sources plus a much larger general-English slice than run 1. Realized token counts below are read from this run's own tokenization stamp files (real post-clean/dedup yield, not the pre-dedup budget estimate — those are known to overstate the real yield on templated legal text):
| Source | HF dataset | Real tokenized tokens |
|---|---|---|
| US case law | HFforLegal/case-law | 0.76B |
| SEC filings (PleIAs) | PleIAs/SEC | 0.83B |
| Educational web (fineweb-edu sample) | HuggingFaceFW/fineweb-edu | 0.93B |
| SEC material contracts | chenghao/sec-material-contracts | 3.60B |
Real tokenized total across fixed sources: 33.02B distinct tokens (excludes the rotating general-web batches above).
Held out of training, same as every other build in this project:
coastalcph/lex_glue, casehold/casehold.
| Initialization | warm-started from data_125m_new/checkpoints/base (run 1's final base weights), NOT random |
| Hardware | 8× NVIDIA B200 GPUs, on-prem |
| Distribution | 8-way DDP, bf16 autocast, SDPA/flash attention, torch.compile |
| Target tokens | 1500B (~11,919 tokens/param) |
| Global batch | 524,288 tokens |
| Optimizer | AdamW, β=(0.9, 0.95), weight decay 0.1, grad-clip 1.0 |
| LR schedule | 0.0006 → 6e-05, Warmup-Stable-Decay (96B warmup tokens) |
python
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizertok = AutoTokenizer.from_pretrained("DeependraVerma/slm-125m-ultimate-base")model = AutoModelForCausalLM.from_pretrained("DeependraVerma/slm-125m-ultimate-base", torch_dtype=torch.bfloat16)prompt = "The plaintiff filed a motion for summary judgment, arguing that"inputs = tok(prompt, return_tensors="pt").input_idsout = model.generate(inputs,max_new_tokens=150,do_sample=True,temperature=0.8,top_p=0.95,eos_token_id=tok.convert_tokens_to_ids("<|eos|>"),pad_token_id=tok.convert_tokens_to_ids("<|pad|>"),)print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))
For browser/edge inference (transformers.js, no server), use the fine-tuned
ONNX export: DeependraVerma/legal-slm-125m-ultimate-sft-onnx.
bibtex
@misc{verma2026legalslm125multimate,author = {Deependra Verma},title = {legal-slm-125M-ultimate: A Warm-Started, Markup-Cleaned 125M-Parameter Legal and Financial Language Model},year = {2026},url = {https://huggingface.co/DeependraVerma/slm-125m-ultimate-base},note = {Code: https://github.com/DeependraVerma/legal-slm-125M}}
Deependra Verma — Generative AI Researcher / AI Engineer. GitHub · Hugging Face
MIT — see LICENSE in the source repo. This is a research artifact, not a source of legal or financial advice.
| SEC EDGAR filings (full) | TeraflopAI/SEC-EDGAR | 14.57B |
| Caselaw Access Project | common-pile/caselaw_access_project | 0.63B |
| Code (permissive) | codeparrot/github-code-clean | 0.78B |
| Long-form books | sedthh/gutenberg_english | 3.71B |
| Annealing supplement | HuggingFaceTB/cosmopedia | 3.03B |
| Wikipedia (en) | wikimedia/wikipedia | 4.20B |
| General web (fineweb-edu, rotating batches) | HuggingFaceFW/fineweb-edu | up to 32×46B budget ceiling (consumed in rotation, not fully materialized on disk at once) |
| Sequence packing | documents packed into contiguous 4096-token windows separated by `< |