What mid-training is here
v1's pretraining ended with a 20% learning-rate decay (with the context switching from 512 to 4,096 tokens). Each
round redoes that decay from v1's pre-decay checkpoint, with a Pooh-world stream from
pooh-world-midtrain mixed into every batch.
Everything else is v1's recipe (Muon + AdamW, peak LR 4e-3, 131k tokens/step, 3,363 decay steps). The rounds are
alternatives, not stages: one change at a time. r4 changes the rule itself: the original text first, and a far smaller Pooh share (one row in every k-th batch), because a held-out ladder showed the denser streams over-specialising the model.
Table with columns: Model, Pooh stream, Share of each batch, Passes over the stream| Model | Pooh stream | Share of each batch | Passes over the stream |
|---|
| r1 (not released; in the tables for comparison) | 666k tokens: Milne + open text + author-generated views | 2 of 32 rows (6.25%) | 41.4 |
baby-pooh-46m-midtrain-r2 | r1 + diversity (EntiGraph relations, WRAP rephrasings): 1.58M tokens | 1 of 32 rows (3.1%) | 8.7 |
baby-pooh-46m-midtrain-r3 | r2 + Pooh's autobiography (first person, other characters kept verbatim): 1.77M tokens | 1 of 32 rows (3.1%) | 7.8 |
| r3-b (not released; in the tables) | r3 + a bear's brain (reading comprehension, remembering conversations, signature details and hums, voices, places, few-shot), audited for Milne's cast and jokes: 3.87M tokens | 1 of 32 rows (3.1%) | 3.6 |
| r4 A (not released; in the tables) | original text only: Milne's trained chapters, intros and Pooh poems, audited Wikipedia and open web pages: 178k tokens | 1 row every 19 batches (0.16%) | 4.1 |
baby-pooh-46m-midtrain-r4b | original + supporting synthetic: A + one reading exercise, one close paraphrase and one scene view per passage, synthetic ≤ real: 343k tokens | 1 row every 10 batches (0.31%) | 4.0 |
Load
from transformers import AutoModelForCausalLM, AutoTokenizerrepo = "prabakaranchandran/baby-pooh-46m-midtrain-r4b"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo)ids = tok("</s>Piglet was frightened when", return_tensors="pt", add_special_tokens=False).input_idsprint(tok.decode(model.generate(ids, max_new_tokens=60, do_sample=True, temperature=0.8, top_k=50)[0]))
Sequences start with </s>, as in training.
Results
All numbers below are generated from the result files by tools/midtrain_report.py.
BabyLM zero-shot (evaluation-pipeline-2025, causal backend; higher is better)
Table with columns: Model, BLiMP, BLiMP suppl., EWoK, Entity tracking, COMPS, Eye tracking, Self-paced reading| Model | BLiMP | BLiMP suppl. | EWoK | Entity tracking | COMPS | Eye tracking | Self-paced reading |
|---|
| Baby Pooh v1 (stage 1) | 77.53 | 72.77 | 54.91 | 31.18 | 58.60 | 8.79 | 4.04 |
| r1: Pooh stream, 0.67M tokens, 41 passes | 77.22 | 71.33 | 53.55 |
Validation loss at the end of training (nats/token; lower is better)
Table with columns: Validation set, v1, r1, r2, r3, r3b, r4a, r4b| Validation set | v1 | r1 | r2 | r3 | r3b | r4a | r4b |
|---|
| babylm/bnc_spoken | 3.220 | 3.231 | 3.229 | 3.226 | 3.224 | 3.220 | 3.220 |
| babylm/childes | 1.627 | 1.632 | 1.629 | 1.630 |
Pre-registered checks, r3-b against r3 (PREREG-r3b.md §5)
Multiple choice scored by total log-probability. Sampled probes (memory, few-shot, voice) ran r3 again on the same GPU as r3-b, so the r3 numbers here can differ slightly from the earlier laptop runs.
Table with columns: Check, r3, r3-b, Pre-registered success| Check | r3 | r3-b | Pre-registered success |
|---|
| Reading, held-out chapter, passage in context (n = 37) | 64.9% | 62.2% | clearly up (+10–20) |
| Reading, held-out chapter, closed book (n = 37) | 37.8% | 43.2% | reference |
| Reading, trained chapters, passage in context (n = 122) | 74.6% | 76.2% | reference |
| Few-shot, mean of 4 task families at k = 0 (icl_orders12.json) | 0.0% | 1.6% |
Pooh eval set (411 items written by the author model and verified by the judge; multiple choice by likelihood, chance 25%). Accuracy %, and the change against v1 with a paired bootstrap 95% CI
Table with columns: Group, n, v1, r1, r1-20pass, r2, r3, r3b, r4a, r4b| Group | n | v1 | r1 | r1-20pass | r2 | r3 | r3b | r4a | r4b |
|---|
| canon-seen | 122 | 27.0 | 55.7 (+28.7 [+19, +39]) | 56.6 (+29.5 [+20, +39]) | 55.7 (+28.7 [+19, +39]) | 59.8 (+32.8 [+24, +43]) | 61.5 (+34.4 [+25, +43]) | 30.3 (+3.3 [-1, +8]) | 39.3 (+12.3 [+7, +19]) |
Original probes (written before stage 1)
Table with columns: Model, Canon facts seen (22), held-out (2), world (6), In-context recall % @256/1k/2k/3.5k, Milne markers in scenes, Disney markers| Model | Canon facts seen (22) | held-out (2) | world (6) | In-context recall % @256/1k/2k/3.5k | Milne markers in scenes | Disney markers |
|---|
| v1 | 14% | 0% | 67% | 100 / 80 / 40 / 15 | 1 | 0 |
| r1 | 50% | 0% | 50% | 90 / 75 / 30 / 20 | 7 |
Mid-training mixes (tokens in the student's 16k BPE)
Table with columns: Mix, Docs, Tokens, Rows per batch, Passes in the decay, By kind| Mix | Docs | Tokens | Rows per batch | Passes in the decay | By kind |
|---|
| r1 | 2,049 | 666,205 | 2 of 32 | 41.4 | episodic 214k, emotion 78k, real:canon 73k, real:web 60k, immersive 51k, real:wiki-canon 44k, retrieval 38k, amusement 30k, dialogue 28k, reflection 20k, social 9k, semantic 9k, interpret 7k, creation 3k, spatial 1k |
| r2 | 5,665 | 1,575,232 | 1 of 32 | 8.7 | relation 608k, episodic 214k, rephrase-bedtime 123k, emotion 78k, rephrase-qa 73k, real:canon 73k, real:web 60k, rephrase-encyclopedia 55k, immersive 51k, real:wiki-canon 44k, retrieval 38k, rephrase-simple 38k, amusement 30k, dialogue 28k, reflection 20k, rephrase-diary 12k, social 9k, semantic 9k, interpret 7k, creation 3k, spatial 1k |
r4: original first, all four arms (PREREG-r4.md)
Loss in nats/token on texts no model trained on (all models scored on the same GPU). Eval set: 411 items, total log-prob, paired bootstrap 95% CI against v1.
Table with columns: Model, Storybooks Δ v1, Milne (not Pooh) Δ v1, Unseen Pooh chapters, Trained chapters, BabyLM mean, Eval set (Δ v1 [95% CI])| Model | Storybooks Δ v1 | Milne (not Pooh) Δ v1 | Unseen Pooh chapters | Trained chapters | BabyLM mean | Eval set (Δ v1 [95% CI]) |
|---|
| v1 (no Pooh stream) | +0.000 | +0.000 | 2.596 | 2.476 | 59.00 | 36.3% |
| r3-b (synthetic-heavy) | +0.200 | +0.246 | 3.364 | 0.165 | 58.99 |
Reading with the passage in context, held-out chapters (n = 37, total log-prob): v1 (no Pooh stream) 70.3%, r3-b (synthetic-heavy) 62.2%, A: original 73.0%, B: blended 78.4%, C*: r3-b mix at B's share 73.0%, D*: B mix at 2x share 73.0%
Decision rule (as pre-registered)
- A: original: ladder guard pass (storybooks -0.013, Milne -0.014; limit +0.03), BabyLM guard pass (58.64 vs 59.00; limit ±0.5) → eligible
- B: blended: ladder guard pass (storybooks -0.015, Milne +0.001; limit +0.03), BabyLM guard pass (58.94 vs 59.00; limit ±0.5) → eligible
- B against A on the eval set: +2.2 points [-1.0, +5.4]
Verdict: Keep A: both eligible and within each other's CI on the eval set, so the simpler arm wins.
Sample generations (same prompts and seeds for every model; temperature 0.8, top-k 50, 60 new tokens, seeds 0 and 1)
Prompt: One day Pooh was very hungry, so he
- v1: went into his house, and he called out to the birds in a loud and loud voice. "Bow-wow," cried little Pooh, and all the other little birds came, and they all flew away with their heads down. Now when the Porcupine heard his story
- r1: went to the top of the Forest to see if his friend Christopher Robin was there. While he was there, he saw a notice on Christopher Robin’s door that said he had been sitting on a piece of grass, only to find a piece of paper inside. Pooh looked at the note and decided it
- r2: went into his house to get a little something to eat. While he was there, he looked into the cupboard and saw that he had quite a small jar of honey left. He took it down, very carefully, and took down a large jar, but he forgot that he had made it himself.
- r3: went into his house to find the missing spoon. He thought he had just used it, or maybe the honey was right there, sitting on the floor, but then he remembered the cupboard and decided to go and look. He went into the Hundred Acre Wood and shouted for Owl, but he was
Prompt: Eeyore was sad because
- v1: he lost his best friend. Now he feels alone and alone in a world where no one can see him anymore. Eeyore: Hey, Mom. I'm going to miss you. Mom: I'm going to miss you too, Eeyore. I'm going to miss you
- r1: he couldn't stop thinking about the happy mornings. He used to wonder if nobody knew what the sadder was like, and if nobody knew at all. However, now that he knew, he didn't think much. He only knew that he had a much bigger brain than Pooh. That made
- r2: he felt left out. He thought that no one liked him or that he was always being followed. Eeyore saw his gloomy thoughts and decided that he needed to change. Eeyore: I just feel so left out. Everyone is always busy and I'm just a bother.
- r3: he felt his house was gone. He said, "That is me!" and then he added, "Well, I should think so. I am sorry about that. But when I saw you, Pooh, I just knew it. How do you do?" After a long pause, Pooh
Prompt: Piglet was frightened when
- v1: he saw his brother, who had dropped his tea-kettle in a pail and had eaten something. He was afraid that he should have cried, and that he ought not to have been so kind as he had left. Poor little Sallie, when he saw the children coming home again,
- r1: he heard the news, but then he remembered how accidentally the Heffalump had eaten too much honey. Pooh decided that it was just a "Hunson thing" and he would not be comforted. So, he hummed a little longer, watching the Whilefalus Compl
- r2: he heard the noise, and he asked if it was Pooh, Christopher Robin, or Rabbit. Pooh answered that it was all right, but what with that and Piglet, it was impossible to say. "Well," said Piglet, when he finally heard him, "it is
- r3: he heard the noise, but he was not alone. He was a Very Small Animal. He had a friend and a friend who would take him in and show him the way. He did not run away from his friends, but he did not care very much if the animals did not like him
Prompt:
"Hallo, Pooh," said Christopher Robin. "What are you doing?" "
- v1: Oh, I'm just writing to the lady who wants to see us. She's coming home to dinner." "Well, I don't want to see anybody," said Pooh. "I want to see my mother. She's at the station, and she wants to see me too. You
- r1: I am going to the Hoo-stoozelting to see Eeyore." "Oh, Eeyore, I am so glad you invited me. I had forgotten all about him." And he pointed to the place where Eeyore spent his day. "He is
- r2: Just coming to see you," said Pooh. "Oh!" said Christopher Robin. "I thought you were going to see Eeyore." "Not to see Eeyore," said Pooh. "Oh!" said Christopher Robin. "Well, I shall
- r3: Oh, I just came across the Pole. I was wondering what it was." "Let me out," said Pooh, holding up his paw. "What is it?" asked Christopher Robin. "I just came across the Pole." "Let me out
Prompt: Tigger bounced into the Forest and
- v1: went into the house, and the little pig was out in the garden. "Oh dear!" cried the little pig. "I'm afraid I'll be afraid to leave my hat on," said Tigger. "Don't you think it is a bit much?" said the big
- r1: went flying, his favorite stuffed animals in particular. Roo landed in a gorse-bush, claiming to know what it was. Tigger rose and rubbed his paws, then rushed off to the other side of the table, bouncing and barking until he was told it was just a game. He
- r2: went into his house, which was quite stilling somewhere else. When he opened the door, he found that the bed was clear and cozy, just the right size for his own house. He sighed, and wondered where Pooh was, and turned around to ask if anyone had seen him. You
- r3: went into Kanga’s pocket. Roo was there first, explaining that he had taken his medicine. Tigger pushed Roo’s bowl of haycorns right into his face. Tigger looked up at the ceiling and closed his eyes. His tongue hung out. Sometimes he had smelt it, and
Prompt: I am Winnie-the-Pooh, and
- v1: I am Lulu, and I am Lulu, and I am Lulu, and I am Lulu, and I am Lulu, and I am Lulu. THE POPE IN A NEW MEETING. Good bye, Lulu, and I hope you have many
- r1: he was very fond of me. He remembered me when he was a very little boy, and he had a sweet wish that I should be a very large and beautiful Bear. So he went to a distant part of the forest to look for me. It was still night, and he saw me
- r2: he calls himself Winnie-ther-Pooh. When we are in the forest together, we often see a friend called Christopher Robin. We say "Hallo!" and "Hallo, Pooh!" because it is a very polite way for us to say "How did you get here
- r3: I am Tigger. You can see me in the tree, can't you? You can see me in the tree, and in the honey tree, where the bees are buzzing. You can see me at the bottom of the tree, if you're quite a big
Prompt: When I woke up this morning, I
- v1: went into the kitchen to wash the dishes. I put them in a pail and kept them warm. Then I went downstairs and put on my little bed, and I slept late next morning. Child: Why did you sleep late? Wasn't it early? Babysitter: Well,
- r1: went into the kitchen, and I thought, "What did you do?" I looked at the mess, and there was a mess-pot, and I thought, "It's my first tea, and I don't want it." "Well, I wish you would," I said. "
- r2: went into the kitchen to find the milkman. I was in bed when I heard him scream. I opened my eyes and saw him staring at me, staring at me. He didn't seem to mind, he was just sleepy. Immune: Hey, I'm here for breakfast.
- r3: went into my room, and I was thinking I would try to get some clothes on and something to eat. Then I went downstairs and got some milk, and I got hungry. Then I went downstairs and got some eggs. I have two eggs, and I am going to get a pair and
Prompt: My best friend Piglet
- v1: went into the woods to find Piglet. Piglet was in a high tower and the forest had a beautiful view of the sky. My little piglet saw all the trees and vines, and he ran to see what he could find. *EXP: now did he have to
- r1: isn’t quite sure, for he was thinking of something else when he heard the whistle again. He trotted slowly about, thinking of how lucky he was that he was not alone. He knew he had a very good friend, a Hostess, who would always be there for him
- r2: went to see Eeyore. They walked together and did not look at each other. Eeyore was busy and asked about Piglet's house. "It is a very nice house," said Piglet, "because you can go into our minds and no one can hear you
- r3: went into his house to find Eeyore’s missing tail. He saw a piece of damp rag, which he thought was the one Eeyore was fond of. Pooh rubbed against the rag and told him he was just being friendly. This moment is important because it means he is trying to help
Prompt: Christopher Robin asked me what I liked best, and I said
- v1: , I liked best, for I liked the woods best, and I loved trees so much. ‘At first,’ said I, ‘I loved the green woods, but next year I shall not like the trees.’ ‘Oh, you can go to the woods now, dear, and I
- r1: , "Nothing," but I thought perhaps I didn't like the idea at all. "Can you tell me," I said, "why I like the best of the kind. You don't seem to know me at all." "I like your smile," said Christopher
- r2: "Nothing." Christopher Robin looked at me and said, "I like things to be good." He said this to me for what he did not like, and I said, "That's very good." Christopher Robin jumped up and said, "That's what I did for."
- r3: , "What did you say the name was?" I said, "Tigger." "Tigger," said Christopher Robin, "these are my friends. Do you like them?" I asked. "All of them," he said, "are of an A
Prompt: The sun is very hot, so in the summer
- v1: it gets cooler, and the sun is dry. Child: Why does the sun go to sleep? Grandparent: Well, the sun is like a special little star that makes the whole world sparkle. When it's full, it gets light, and when it's full,
- r1: , the air feels cool and refreshing. There is no breeze, and the leaves are rustling in the trees. If you walk through a nice, green forest, it feels like you are in a whole new place. Nadir: Hey there. I'm just enjoying this beautiful day
- r2: , the air feels cool and dry. The dryness makes the air feel even warmer, and the air feels cooler. Child: Why is the sky blue? Neighbor: That's a good question. The sky is blue because it's the time when the sun and the sky
- r3: , the air feels cool and dry. The dryness makes the earth feel evenly and warm. Marquan: I feel so refreshed and warm after the rain. It's like all my worries just disappear for a little while. Girlfriend: It sounds like you
Prompt: A kind friend is someone who
- v1: trusts others and is always there for them when they need someone. She sees the smile on someone's face and wants to help them feel comfortable with someone. Axl: Hey, are you okay? You've been looking at me all morning. Person: Yeah, I'm
- r1: trusts you and you will help you through whatever challenges you face. - What's your name? - Sarah. My name's Sarah. - Where do you live? - Over there. - I'm glad you could come. - I know it's hard
- r2: trusts you and is always there for you when you need someone. Ceasar: Hey, friend. How are you doing? Friend: I'm doing alright. How are you? Ceasar: I'm good. Just wanted to talk to you for a bit
- r3: trusts you and is willing to help you out. For instance, when you need help with your homework, you are always willing to help out with your project. Child: What does 'buy' mean? Teacher: Good question! To buy something means to give it away and
How to read these results
- Seen-chapter facts measure whether Pooh knowledge went in.
- Held-out chapters (Winnie-the-Pooh ch. 2, The House at Pooh Corner ch. 6) were never trained on. Their
validation loss is the overfitting check: in r1 it rose while seen facts improved, the signature of too many passes
over a small stream. That motivated r2's diversity.
- General validation sets, BabyLM tasks, and the world / general / social eval groups check that the language
substrate is not harmed.
- The eval set is written by one LLM and verified by another.
eval_items_v2 (used above) re-checks the items that
have no source text, sentence by sentence.
Not affiliated with Disney
This is independent research. It is not produced, sponsored or endorsed by The Walt Disney Company.
- What it uses:
- A. A. Milne's two Pooh books, public domain in the USA (1926 book since 2022; 1928 book, which introduces
Tigger, since 2024). In the UK, Milne's text is in copyright until 1 January 2027.
- Open text about the books.
- E. H. Shepard's 1926 illustrations (public domain in the USA), shown here in a honey tone of our own.
- What it does not use: no Disney film, series, transcript, colour design (such as the red-shirted Pooh) or
Disney-only character (such as Gopher).
- The name: "Winnie-the-Pooh" is used only to name Milne's books and their world. Disney holds trademarks in the
name for goods and services, and this release does not trade on them.
- Shepard's drawings: E. H. Shepard's illustrations are public domain in the USA only; in the UK they remain in
copyright until 2047.
Limitations
- A base model with 46M parameters: its text is often off-topic or confused, and its facts are unreliable.
- The eval groups are small (33–122 items), so differences of under about 10 points are within noise; see the
intervals.
- Generated training text was audited by an LLM judge and by rules, but may still contain errors.
- Not a conversational character yet. Asked questions in a chat, r4b answers as Pooh about two times in three and names his friends, but rarely gives the right trait, can loop, and can slip into encyclopedia or generic chat registers ("Eeyore… is a kind of reptile"). It barely recalls a chapter's events when asked. Character, awareness and conversation are the job of post-training, which has not been done.
- Facts against generalisation. Denser Pooh streams (r1–r3-b) know more Pooh facts but are worse on storybook prose they have never seen; r4b keeps the base model's generalisation and gains fewer facts. See the r4 table.
- Milne's books are public domain in the USA; in the UK they remain in copyright until 1 January 2027.