GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
miilg
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
miilg
Model Tree
Qwen/Qwen2.5-0.5B
Input Modalities
Output Modalities
Supported Functionality
Qwen/Qwen2.5-0.5B with its tokenizer and (tied) embedding matrix pruned to the tokens that caveman English actually
uses. Caveman English is a closed-vocabulary register of English: 961 lowercase base-form words, frequent names
Capitalized, digits and . , ? ! — e.g. "mammoth be big animal with long hair."
Kept: every token that occurs at least once (m = 1) in a 9.2 MB caveman corpus (English Wikipedia rewritten by
claude-haiku-5-5), all special tokens, the 256 byte-level characters and the closure over BPE merges. Ids are renumbered
contiguously; the embedding is sliced by index (no retraining) and padded to a multiple of 64.
| original | pruned | |
|---|---|---|
| vocabulary / merges | 151,643 / 151,387 | 6,244 / 5,988 |
| parameters | 494.0 M | 363.5 M (−26.4%) |
| bf16 checkpoint | 1000 MB | 727 MB |
| held-out caveman bits per byte | 1.2846 | 1.2279 |
| ordinary English bits per byte | 0.970 | 2.102 |
The train corpus is tokenized exactly as by the original tokenizer. Any text still round-trips (byte-level alphabet), but text outside the domain is split into many more tokens and modelled much worse. The lower bits per byte on caveman text comes from renormalising the softmax over fewer tokens, not from a better model.
python
from transformers import AutoModelForCausalLM, AutoTokenizerrepo = "miilg/qwen2.5-0.5b-caveman"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16")x = tok("mammoth be big animal with long hair. people", return_tensors="pt")print(tok.decode(model.generate(**x, max_new_tokens=30, do_sample=False)[0]))Code, corpus (CC BY-SA 4.0), report and the full threshold sweep: https://github.com/miilv/prune-to-caveman (Skoltech, Modern Methods and Algorithms of Generative AI, Homework 1, Ilia Mikhalchuk).