Model details
Table | |
|---|
| Parameters | 348,342,912 (~348M) |
| Architecture | Decoder-only transformer, pre-norm, RMSNorm, SwiGLU MLP, RoPE |
| Hidden size | 1152 |
| Layers | 22 |
| Attention heads | 18 query / 6 KV (grouped-query attention), head_dim 64 |
| Context length | 2048 |
| Vocabulary | 32,000 (tbb-32k-v2, tied embeddings) |
| RoPE theta | 10,000 |
| Training tokens | 22.70B |
| Precision | fp16 with fp32 master weights (final phase) |
Evaluation
EleutherAI lm-eval-harness, 0-shot, complete test sets. acc_norm for
HellaSwag/ARC/OpenBookQA/PIQA, acc for WinoGrande/MMLU. Same harness and settings as the
published 100M V3 numbers, so the comparison is like-for-like (base vs base).
Table with columns: Benchmark, 350M V3 base, 100M V3 base, Supra2-100M-Base| Benchmark | 350M V3 base | 100M V3 base | Supra2-100M-Base |
|---|
| ARC-Easy | 56.6 | 54.7 | 47.8 |
| ARC-Challenge | 33.3 | 30.1 | 24.8 |
| HellaSwag | 35.9 | 32.6 | 36.0 |
| OpenBookQA | 34.6 | 34.0 | 32.0 |
→ 5/7 vs the 100M V3 base, losing only WinoGrande (−1.5) and MMLU (−1.2).
→ 7/7 vs Supra2-100M-Base (HellaSwag 35.9 vs 36.0 is a 0.1 tie at best).
Benchmark-train-split disclosure
The final data mix contains two small sets built from official train splits of
benchmark datasets:
sci-qa-train (0.5%) — SciQ, ARC-Easy/Challenge train, OpenBookQA train + core factshellaswag-train (1.2%) — HellaSwag train, SWAG train
Evaluation uses the test/validation splits, which these never touch, and this is standard
practice for small-model training. But the 100M V3 baseline above was trained without them,
so the ARC / OpenBookQA / HellaSwag margins are not a like-for-like recipe comparison. Stated
here rather than buried: if you want a clean comparison, weight those three tasks accordingly.
Training
Three phases, on rented and owned hardware:
- Cloud pretrain — ~20B tokens on a rented 8× RTX 5090 node (Clore.ai), bf16/fp8,
torch.compile, DDP. Cost about $90.
- Local extension — continued on 2× Tesla V100 (16GB) in fp16, WSD schedule, sequence
length 2048, micro-batch 2 × grad-accum 32 × 2 GPUs = 262,144 tokens/step.
- Fact anneal — two final passes on a knowledge-dense mix (below), 1.23B tokens, LR
annealed 5e-5 → 1e-5.
The frozen-norm episode
For its first ~20B tokens this model trained with all 45 RMSNorm weights frozen at exactly
1.0. The cause was precision, not the optimiser: --precision bf16 stored the weights in
bf16 with no fp32 master copy. bf16 has 7 mantissa bits, so near a value of 1.0 the
smallest representable step is ~0.0078, while an Adam update at lr 1e-4 is ~1e-4 — every norm
update rounded straight back to 1.0. Other weights (magnitude ~0.02) have a ~50× finer ULP
there and trained normally, so loss fell and nothing looked wrong.
It was found by diffing two consecutive checkpoints: 111/111 non-norm tensors changed,
0/45 norms. Switching to fp16 (which keeps fp32 masters) unfroze them.
They never fully recovered. Measured mean |w − 1| across the RMSNorm weights:
| model | mean·|w−1| |
|---|:--:|
| 100M V3 base (never affected) | 0.697 |
| 320M V2 base (never affected) | 0.694 |
| 350M V3 base (this model) | 0.017 |
After 5.5B tokens of healthy fp16 training the norms had moved 2.5% of the distance a
normally-trained model's travel. So this model does its work essentially without learned
per-channel scaling — and still beats a 100M trained with it. Treat the results below as a
lower bound on what this recipe can do.
Training data
Three phases, each with its own mix. Every source was checked against measured token counts
(tools/dataset_true_tokens.json) rather than a filesize/4 proxy, which misjudges dataset
size by 0.07x-5.2x and once hid a source running at 6.9 epochs.
Phase 1 - cloud pretrain (~20B tokens), 20 sources. Web-heavy foundation.
Table with columns: Source, Share| Source | Share |
|---|
fineweb-edu-100bt | 49.3% |
gutenberg-clean2 | 8.0% |
bookcorpus-clean | 6.1% |
dclm-clean | 5.6% |
wiki-dense | 4.2% |
wikipedia-english | 3.8% |
Phase 2 - local extension, 18 sources. Shifts toward reference and synthetic-educational
text as the LR decays.
Table with columns: Source, Share| Source | Share |
|---|
wiki-dense | 13.9% |
wikipedia-english | 11.8% |
cosmopedia-stanford | 10.7% |
cosmopedia-auto-math | 10.7% |
qa-plain | 9.6% |
dclm-clean | 8.5% |
Phase 3 - fact anneal (1.23B tokens), 22 sources. Knowledge-dense final pass. Top 12:
Table with columns: Source, Share| Source | Share |
|---|
wiki-dense | 12.9% |
cosmopedia-openstax | 12.9% |
cosmopedia-stanford | 8.9% |
qa-plain | 7.9% |
wikipedia-english | 7.0% |
gutenberg-clean2 | 5.0% |
Sources beginning cosmopedia-, qa-, facts-, reasoning- and capability- are
synthetic/distilled sets generated locally; the rest are public corpora (FineWeb-Edu, DCLM,
Wikipedia, Gutenberg, BookCorpus, TinyStories). Narrative and commonsense sources total 21.1%
of the anneal.
Note on the anneal share. This model spent roughly a quarter of its training on the
anneal-style diet, against ~2% for the 100M V3. That is a large share for what is meant to be
a finishing pass, and it is stated here because it is a plausible contributor to the
benchmark profile - though this model also carries the frozen-norm bug described above, so
the two cannot be cleanly separated.
Limitations
- Arithmetic is weak. This is a base model with no math tuning; use the math variant.
- MMLU and WinoGrande lag the 100M V3 base. The knowledge-dense anneal did not lift MMLU.
- Repetition loops on greedy decoding beyond ~40 tokens, as is normal at this scale.
- Hallucinates confidently — it will state a wrong country for a landmark under sampling.
- No instruction tuning. It continues text; it does not follow instructions or chat.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("nkthebass/tinybrainbot-350mV3-base")
t = AutoTokenizer.from_pretrained("nkthebass/tinybrainbot-350mV3-base")
ids = t("The capital of France is", return_tensors="pt")
print(t.decode(m.generate(**ids, max_new_tokens=30)[0]))