How it is built
A single-GPU recipe (RTX 3090 Ti):
- From-scratch pretraining on open, permissively-licensed educational English text plus our
own generated reasoning and arithmetic data. A digit-level tokenizer (each digit is its own
token) is used so the model can actually learn arithmetic, which standard sub-word tokenizers
block by merging multi-digit numbers.
- Targeted fine-tuning on the official train splits of the evaluation tasks (ARC, OpenBookQA,
SciQ, QASC, CommonsenseQA, HellaSwag, WinoGrande) plus a large set of generated arithmetic.
Train splits only, with no test contamination.
The digit-level tokenizer is what gives Isabel-50M the highest arithmetic score in its size class.
Architecture
Table | |
|---|
| Type | Decoder-only transformer |
| Hidden size | 512 |
| Layers | 9 |
| Heads | 8 |
| Vocab | 32,000 (our own digit-level byte-level BPE) |
| Context | 1024 |
Evaluation (0-shot, full test sets)
Table with columns: Benchmark, Isabel-50M| Benchmark | Isabel-50M |
|---|
| HellaSwag (acc_norm) | 27.1 |
| ARC-Easy (acc_norm) | 43.8 |
| ARC-Challenge (acc_norm) | 23.5 |
| PIQA (acc_norm) | 57.3 |
| ArithMark-2 (acc) | 42.4 |
| Average | 40.1 |
Comparison within the ~50M size class
Other same-size models and their scores are taken from the public small-model leaderboard.
Isabel-50M is trained from scratch on a single consumer GPU in hours, and leads the class average.
Table with columns: Model, Params, Avg, HellaSwag, ARC-E, ARC-C, PIQA, Arith| Model | Params | Avg | HellaSwag | ARC-E | ARC-C | PIQA | Arith |
|---|
| Isabel-50M | ~54M | 40.1 | 27.1 | 43.8 | 23.5 | 57.3 | 42.4 |
| Supra-1.5-50M-base-exp | 51.8M | 39.0 |
Isabel-50M leads the class on average and has the highest arithmetic score, driven by its
digit-level tokenizer. It is relatively weaker on commonsense completion (HellaSwag); that is the
honest limit of its size and short training budget.
Citation
@misc{isabel_2026, title = {Isabel-50M: A Tiny From-Scratch Language Model for the Edge}, author = {Malios Dark}, year = {2026}, note = {Ideoa Labs}}