perletter

dot-125m

README

License: apache-2.0

Model details

Table

Parameters	133.7M
Architecture	Llama-3-style decoder-only, GQA, RoPE, RMSNorm, SwiGLU, tied embeddings
Layers / heads / KV heads	12 / 12 / 4
Hidden size	960
Context length	512
Vocab size	16,384 (byte-level BPE, trained from scratch on the training split)
Training tokens	2.0B (4 epochs × 500M-token corpus)
Training data	FineWeb-Edu (`sample-10BT`), quality-filtered + exact/near-deduplicated, English only
License	Apache 2.0

Benchmarks

Compared against GPT-2-small (124M, ~10B training tokens) — see full write-up for methodology.

Bits-per-byte (primary metric, tokenizer-fair comparison), on a held-out test split:

Table with columns: bpb, ppl
	bpb	ppl
Dot-125M	1.0142	20.64
GPT-2-small	1.0281	27.19

lm-evaluation-harness:

Table with columns: task, Dot-125M, GPT-2-small
task	Dot-125M	GPT-2-small
arc_easy (acc)	48.23%	43.81%
hellaswag (acc_norm)	30.86%	31.14%
piqa (acc)	61.43%	62.89%
winogrande (acc)	49.57%	51.62%
lambada_openai (acc)	23.02%	32.56%

Mixed on the individual benchmark tasks (stronger on arc_easy, weaker on lambada_openai's long-range prediction — expected given the token/context budget: Dot-125M saw 500M unique tokens across 4 epochs at 512 context vs. GPT-2's ~10B tokens single-pass at 1024 context), but wins on the primary bits-per-byte metric.

Quantization (GGUF, via llama.cpp): Q8_0 stays within 0.01% bpb of full-precision f16. Q4_K_M is available but falls back to a different quant scheme for most tensors (this model's hidden size isn't a multiple of 256, the k-quant block size) — still only ~0.17% bpb degradation vs. f16, but not "true" Q4_K_M. Q8_0/Q4_0/Q5_0/Q5_1 are the quant types this model size supports natively.

Intended use

Research, experimentation, and demonstration of from-scratch small-LM pretraining. Not instruction-tuned — do not expect chat-assistant behavior out of the box. Not suitable for production use requiring factual reliability, safety filtering, or instruction following without further fine-tuning.

How to use

Transformers (safetensors):

python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("perletter/Dot-125M")
model = AutoModelForCausalLM.from_pretrained("perletter/Dot-125M")

inputs = tok("The history of the internet", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=50)
print(tok.decode(out[0], skip_special_tokens=True))

llama.cpp (GGUF):

bash
llama-cli -m model-Q8_0.gguf -p "The history of the internet" -n 50

Limitations

Small model, small training budget — general knowledge and reasoning are limited compared to larger contemporary models.
English only.
Base model only — no safety fine-tuning, no RLHF, no instruction-tuning. It will complete harmful, biased, or false text if prompted toward it, the same as any unaligned base LM.
512-token context window.

License

Apache 2.0 — the model weights are freely available for any use, including commercial, with no attribution requirement beyond the license notice. See LICENSE.

The training code/pipeline used to produce this model is not included in this release.

Citation

markdown
@misc{dot125m2026,
  title  = {Dot-125M},
  author = {Perletter, part of Chirping Waves Limited},
  year   = {2026},
  url    = {https://perletter.com}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

perletter

Model Tree

Base

this model

Input Modalities

Text

Output Modalities