prabakaranchandran

baby-pooh-46m

README

License: apache-2.0

Summary

Architecture: decoder-only Transformer, 12 layers, width 512, 8 query / 4 key-value heads (GQA), QK-norm, RoPE (θ = 10,000), SwiGLU (1,536), RMSNorm, tied embeddings. 46,151,680 parameters. The block is exactly Qwen3's, so the model loads as Qwen3ForCausalLM. The export is checked for exact logit parity with the training code (max |Δlogit| = 0.0 in transformers 4.51 and 5.19).
Data: pooh-babylm-mix-220m, 220.1M tokens (133.7M words):
- the BabyLM 2026 Strict corpus (100M words)
- TinyDialogues, SODA and FineWeb-Edu subsets (about 16% synthetic)
- A. A. Milne's four Pooh books, minus two held-out chapters
Tokenizer: BabyLM's 16,384-token BPE. Sequences start with </s> (as in training), not <s>.
Training: 10 epochs ≈ 2.2B tokens. Muon (hidden matrices) + AdamW (embedding, norms), peak LR 4e-3 (best of a 3-point sweep; Muon beat an AdamW control by about 0.75 nats at 2.5% of the budget). Warmup–stable–decay schedule with a 20% linear decay; 512-token context, switching to 4,096 for the decay. ~131k tokens/step, 16,815 steps, one H100, about 70 minutes.
Training curves: W&B project babypooh, run qgrij0oi.

Results

BabyLM evaluation-pipeline-2025 (zero-shot, causal backend). All three models are scored by us with the same pipeline and data (BabyLM 2026 evaluation sets; EWoK = fast set). Higher is better.

Table with columns: Model, Params, BLiMP, BLiMP suppl., EWoK, Entity tracking, COMPS, Eye tracking, Self-paced reading
Model	Params	BLiMP	BLiMP suppl.	EWoK	Entity tracking	COMPS	Eye tracking	Self-paced reading
Baby Pooh v1	46M	77.53	72.77	54.91	31.18	58.60	8.79	4.04
BabyLM baseline GPT-2 Small (100M words)

Not a like-for-like comparison: Baby Pooh saw 133.7M words (above BabyLM's 100M) and about 16% synthetic text. GLUE fine-tuning: pending.

Validation loss (nats/token) on held-out text, final model:

Table with columns: Source, Loss, Source, Loss
Source	Loss	Source	Loss
SODA	1.491	Switchboard	2.173
TinyDialogues	1.581	Held-out Milne ch. (WtP 2)	2.560
CHILDES	1.627	Held-out Milne ch. (HPC 6)	2.622
Simple Wikipedia	2.136	OpenSubtitles	2.982

Probes of Pooh knowledge and memory (tools/probe_pooh.py; multiple choice by likelihood):

Canon facts from chapters it trained on: 14% (chance 25%). Everyday-world facts: 67%. The model has Milne's style, not his facts: each chapter was 0.05% of the data. Adding Pooh-world knowledge is the next stage.
Verbatim memory: 6.7% token match when continuing passages from trained chapters, with no exact continuations. No memorisation of the books.
In-context recall of a name planted earlier in a story: 100% / 80% / 40% / 15% at 256 / 1,024 / 2,048 / 3,584 tokens (chance 20%).

Use

python

from transformers import AutoModelForCausalLM, AutoTokenizerrepo = "prabakaranchandran/baby-pooh-46m"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo)ids = tok("One morning, Piglet", return_tensors="pt").input_idsout = model.generate(ids, max_new_tokens=60, do_sample=True, temperature=0.8, top_k=50)print(tok.decode(out[0], skip_special_tokens=True))

Checkpoints:

main (and the tag v1) is the final model.
Branches chck_<N>M hold the model after N million words (counting repeats), on BabyLM's schedule: every 1M up to 10M, every 10M up to 100M, then every epoch (133M, 267M, …, 1203M). Example: AutoModelForCausalLM.from_pretrained(repo, revision="chck_10M").

Limitations

No Pooh world knowledge yet (see the probes), and no stable identity: in generations Pooh can turn into "Mr. Bear".
Not instruction-tuned or safety-tuned. It continues text; it does not follow instructions.
Format leakage from the corpus: occasional Mom:/Child: turns (TinyDialogues), CHILDES codes, and Gutenberg [Illustration…] markers.
Small and English-only.
Copyright: the Milne texts in the training data are public domain in the US but under copyright in the UK and EU until 1 January 2027.
Not for impersonating the Disney characters or brand.

Not affiliated with Disney

This is independent research. It is not produced, sponsored or endorsed by The Walt Disney Company.

The Pooh text in the training data is A. A. Milne's books as published in the 1920s, which are public domain in the USA. In the UK, Milne's text is in copyright until 1 January 2027.
The model uses no Disney film, series, transcript, design or Disney-only character.
"Winnie-the-Pooh" names Milne's books and their world. Disney holds trademarks in the name for goods and services, and this release does not trade on them.
E. H. Shepard's illustrations are public domain in the USA only; in the UK they remain in copyright until 2047.

Citation

Dataset: prabakaranchandran/pooh-babylm-mix-220m. Evaluation: Choshen et al. (2026), BabyLM Turns 4, arXiv:2602.20092; the BabyLM evaluation-pipeline-2025.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

prabakaranchandran

Model Tree

Base

this model

Input Modalities

Text

Output Modalities