Usage
from transformers import AutoModelForCausalLM, AutoTokenizer repo = "yzdvai/tinystories-llama-64m"tokenizer = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo) inputs = tokenizer("Once upon a time, a little robot found a book.", return_tensors="pt")output = model.generate(**inputs, max_new_tokens=200, do_sample=True, temperature=0.8, top_k=40, pad_token_id=tokenizer.pad_token_id)print(tokenizer.decode(output[0], skip_special_tokens=True))
No BOS token is used. Each story ends with <eos>, so generation stops by itself.
Architecture
Table | |
|---|
| Total parameters | 64,365,440 |
| Layers | 13 |
| Hidden size | 640 |
| Attention heads | 10 (head size 64) |
| KV heads | 5 (grouped-query attention, 2 query heads per KV head) |
| Intermediate size | 1728 (SwiGLU) |
| Vocabulary | 8,192 (byte-level BPE) |
| Context length | 512 tokens |
| Other | RoPE positions, RMSNorm (pre-norm), no biases, tied input and output embeddings |
Where the parameters are: SwiGLU 43,130,880 (67%), attention 15,974,400
(25%), embedding 5,242,880 (8%).
Why these choices
- Llama-style block. RoPE, RMSNorm, SwiGLU and grouped-query attention are what most open
models use today. It is a plain
LlamaConfig, so it loads with AutoModelForCausalLM and needs
no custom code.
- Size near the 60M floor. The assignment asks for 60-100M. My real compute was
90 minutes on 2 x Tesla T4, and for that compute the scaling rule says a much
smaller model is optimal (see below). So I stayed close to the floor instead of going bigger.
- Wider and shallower (640 wide, 13 layers) than the lab's 512 wide, 24 layers. Fewer, wider
layers run faster on a GPU, which means more tokens in the same time.
- Small vocabulary (8,192). TinyStories uses a few thousand distinct words. A small
vocabulary keeps the embedding at 8% of the model, so the parameters go
to the layers.
- Context 512. A story is about 210 tokens, so a whole story fits with room to spare.
Dataset
Table | |
|---|
| Source | roneneldan/TinyStories, train split (Eldan and Li, 2023) |
| Language | English |
| Licence | CDLA-Sharing-1.0 |
| Stories sampled | 900,000 (shuffled, seed 42) |
| Stories kept | 791,092 |
Why TinyStories. With about 9% of the planned token budget, a broad corpus such
as Wikipedia would give word salad. TinyStories was built so that small models can learn to write
coherent text, so it is the corpus where a short run still shows real learning.
Preprocessing
- Straighten curly quotes and dashes, collapse whitespace.
- Drop stories under 100 characters (88), stories with non-ASCII
characters, which in this dataset are mostly encoding damage (55,164),
and exact duplicates (53,656).
- Split by whole story, before tokenizing, so no story is in two splits.
- Train a byte-level BPE tokenizer on training stories only (first 200,000).
- Tokenize, add
<eos> after each story, join, and cut into blocks of 512 tokens with no overlap.
Table with columns: Split, Stories, Tokens, Blocks of 512| Split | Stories | Tokens | Blocks of 512 |
|---|
| Train | 775,270 | 163,954,618 | 320,223 |
| Validation | 7,911 | 1,677,349 | 3,276 |
| Test | 7,911 | 1,665,592 | 3,253 |
Scaling-law calculations
Planning rule from the lab (a simplified Chinchilla): D = 20 N and C = 6 N D.
Table | |
|---|
| Parameters N | 64,365,440 |
| Planned tokens D = 20 N | 1,287,308,800 |
| Planned compute C = 6 N D | 4.97e+17 FLOPs |
| Tokens actually processed | 113,737,728 (8.8% of plan) |
| Scored next-token targets | 113,515,584 |
| Actual compute, 6 N D | 4.39e+16 FLOPs |
| Passes over the training tokens | 0.69 (no block was seen twice) |
| Full plan at the measured speed | about 17 hours on the same hardware |
Read the other way: for the compute I actually used, the same rule gives an optimal size of
N = sqrt(C / 120) = 19M parameters. So this model is larger than its training budget
justifies.
What I saw. The loss was still falling at the end, but slowly: validation went from 1.93 at
30M tokens to 1.65 at 62M and 1.51 at 114M. Each doubling of data bought less. The 20:1 ratio was
measured on broad web text. TinyStories is much simpler and more repetitive, so the model runs out
of new things to learn from it sooner. The rule sets the plan, but how many tokens are worth
processing also depends on how hard the data is. On this corpus I expect the remaining 91% of the
plan to bring a modest gain, not a large one. I did not test that.
Training setup
Table | |
|---|
| Hardware | 2 x Tesla T4 (Kaggle), mixed precision (float16) |
| Optimizer | AdamW, betas (0.9, 0.95), weight decay 0.1 on weight matrices only |
| Learning rate | 0.0006 peak, 100 warmup updates, cosine decay to 6e-05 |
| Effective batch | 64 sequences x 512 tokens = 32,768 tokens per update (16 per GPU x 2 GPUs x 2 accumulation) |
| Gradient clipping | 1.0 |
| Training steps | 3,471 optimizer updates |
| Training duration | 90 minutes (21,062 tokens per second) |
| Seed | 42 |
Training was limited by time, not by steps: the cosine schedule follows the clock, so the learning
rate reaches its floor exactly when the time budget ends.
Evaluation
Held-out next-token cross-entropy (nats) and perplexity, on stories the model and the tokenizer
never saw.
Table with columns: Split, Loss, Perplexity, Scored tokens| Split | Loss | Perplexity | Scored tokens |
|---|
| Validation (full) | 1.521 | 4.57 | 1,674,036 |
| Test (full) | 1.526 | 4.60 | 1,662,283 |
Before training, the loss on the validation subset was 9.14 (a uniform guess is
ln 8192 = 9.01). After training it was 1.51. Perplexity only
compares with models that use this same tokenizer and data.

Training loss is from one batch every 10 updates (the dark line is a moving average). Validation loss is from a fixed subset of
640 blocks every 100 updates. Every training batch is new data, so the two
curves sit close together.
Example outputs
Sampling with temperature 0.8, top-k 40, fixed seeds. Not cherry-picked.
Prompt: Once upon a time, a little robot found a book.
Once upon a time, a little robot found a book. The book was very tall, with lots of pages. The robot was very happy. It wanted to keep the book and read it all day. The robot tried to open the book, but it was too hard. The robot was sad. One day, the robot saw a big, red ball. The robot wanted to help the ball. The robot had an idea. It called the red ball, "Hello, red ball!" The red ball opened the book. The robot was very happy. The robot and the red ball became good friends. They played together in the park every day. The robot and the red ball became best friends. They were very happy.
Prompt: Lily and her dog went to the park. Suddenly,
Lily and her dog went to the park. Suddenly, Lily saw a little boy crying. She asked him what was wrong. He told her about his dog and his ice-cream. Lily was compassionate and kind. She said, "Don't worry, we can share the ice-cream with your dog." The little boy smiled and they shared the ice-cream. It was a great day at the park.
Prompt: The old tree in the garden had a secret.
The old tree in the garden had a secret. A secret that no one could find. The people in the garden laughed and said they would never find the secret again. They kept the secret and the old tree stayed there forever.
Limitations
- Undertrained. It saw 8.8% of the planned token budget.
- One narrow domain. It only knows simple children's stories. It cannot answer questions,
follow instructions, or write about anything else.
- Synthetic data. TinyStories was written by GPT-3.5 and GPT-4, so the stories repeat the same
names, plots and morals, and the model copies that.
- Weak logic. The grammar is good but the events often make no sense. In the samples above the
book is "very tall", a ball opens the book, and the tree's secret is never told.
- Repetition and thin plots. It repeats itself ("became good friends" and then "became best
friends") and some stories end after two or three sentences.
- English, ASCII only. Other languages and symbols split into bytes and give poor results.
- Single run. One seed, no hyperparameter search. One run does not prove a scaling law.
Credits
Yazeed Alhifzi. The training code was written with Claude. Based on the course lab "From a compute
budget to a 75M-parameter language model". References: Hoffmann et al. 2022 (Chinchilla),
Eldan and Li 2023 (TinyStories).