Architecture
Table with columns: Item, Value| Item | Value |
|---|
| Total parameters | 74,998,272 |
| Layers | 24 |
| Hidden size | 512 |
| Attention heads (query) | 8 (head dim 64) |
| KV heads (GQA) | 4 |
| Intermediate (SwiGLU) size | 1408 |
| Vocabulary size | 8192 (byte-level BPE trained on the training split only) |
| Context length | 512 (trained on 512-token blocks) |
| Other | RoPE positions, RMSNorm, SwiGLU, no biases, tied input/output embeddings |
Reasoning. N = sqrt(C/120) = 75M and D = 20N for C = 6.75e+17 FLOPs; the layer count is solved from the target N. Head dim 64 is the standard width; GQA (4 KV heads for 8 query heads) halves K/V projections and the KV-cache; the SwiGLU width is about 8/3 x hidden so it costs the same as a classic 4x FFN; tied embeddings put 4.2M parameters into depth instead of a second embedding matrix; the 8k vocabulary suits the simple English of TinyStories (a 32k vocabulary would put about a quarter of the parameters into embeddings). I did not run depth/width ablations, so the configuration is reasoned rather than empirically optimised.
Dataset
- Source: roneneldan/TinyStories (training split, shuffled stream), language: English, licence: CDLA-Sharing-1.0.
- Preprocessing: whitespace normalisation and exact deduplication (SHA-256); 600,000 unique stories kept. Stories get an
<eos> token, are concatenated per split and cut into non-overlapping 512-token blocks (last partial block dropped). The tokenizer was fitted on 200,000 training stories only.
- Splits (by whole story, seed 42): train 540,000 / validation 30,000 / test 30,000 stories (90/5/5). Tokens: train 113,610,772, validation 6,324,950, test 6,331,738.
Scaling-law calculations
- Rule of thumb: C ~ 6ND and D = 20N, so N = sqrt(C/120). With C = 6.75e+17 FLOPs: planned N = 75.0M parameters, planned D = 1.50B tokens.
- Actual architecture N = 74,998,272, so the 20N target is 1,499,965,440 tokens.
- Actual training tokens processed: 38,862,848 = 2.6% of the 20N target (0.34 passes over 113,610,772 unique training tokens). Approximate compute 6 x N x tokens = 1.75e+16 FLOPs.
- The model is therefore under-trained relative to the compute-optimal estimate: the budget was limited by GPU time (measured 3,872 tokens/s; the full 20N run would take about 108 hours on this GPU).
Training setup
Table with columns: Item, Value| Item | Value |
|---|
| Hardware | Tesla T4 |
| Precision | bf16 mixed precision (FP32 master weights) |
| Optimizer | AdamW (betas 0.9/0.95), weight decay 0.1, gradient clipping 1.0 |
| Learning rate | peak 0.0006, 11-step linear warm-up, cosine decay to 0 |
| Effective batch size | 128 sequences x 512 tokens = 65,536 tokens/update (micro-batch 16 x gradient accumulation 8) |
| Training steps | 593 successful optimizer updates (0 skipped for FP16 overflow) |
| Training duration | 2.82 h of pure training time |
| Seed | 42 |
Evaluation
Held-out next-token cross-entropy of the final weights on the full validation split and the full test split (the test split was not used for any decision):
Table with columns: Split, Loss (nats/token), Perplexity| Split | Loss (nats/token) | Perplexity |
|---|
| Validation (full) | 2.7590 | 15.78 |
| Test (full) | 2.7623 | 15.84 |
Perplexity is per token of this model's own 8k tokenizer, so it is not comparable with models that use other tokenizers. The untrained model scored 9.112 on the validation subset (a uniform guess gives ln V = 9.011).

Training loss = mean of the last 25 updates; validation loss = fixed 25-batch subset of the validation split, evaluated every 250 updates. Training loss decreased from about 9.12 to 2.77, while the fixed validation-subset loss decreased from about 9.13 to 2.74. Improvement became smaller near the end, and the recorded validation points stayed close to the training curve without a clear sign of overfitting during this run.
Example outputs
Prompt: Once upon a time, a little robot found a book.
Greedy: Once upon a time, a little robot found a book. The little bird was very happy and wanted to play with it. The little bird was happy and said, "I can help you." The little bird was so happy and said, "I'm glad you are safe. I will help you." The little bird was so happy and said, "I'm glad you are safe. I will help you." The little bird was so happy and said, "I will help you. I will help you." The little bird was so happy and said, "I will help you. I will help you." The little bird was so happy and said,
Sampled (T=0.8, top-k 50, top-p 0.95): Once upon a time, a little robot found a book. The little bird was very happy and could not help everyone. But one day, the little bird saw a big, dark tree. The little bird was sad because he knew it was important to be happy. He looked up and looked for the bird, but it was too strong for him. The small bird was happy to be able to play with his friends. He ran to his friends and the bird and the bird were the best of friends. They played with the bird together, and they had a fun day.
Prompt: Tom and his dog went to the park. They saw
Greedy: Tom and his dog went to the park. They saw a big tree with a big tree. They wanted to see the tree. "Look, Ben, a big dog!" Tom said. "I want to play with the ball. It is a big dog. The dog is big and strong. It has a big ball. It is a big dog. The dog is big and strong. The dog is angry. The dog is angry. The dog is angry. The dog is angry. The dog is angry. The dog is angry. The dog is angry. The dog is angry. The dog is angry. The dog is angry. The dog
Sampled (T=0.8, top-k 50, top-p 0.95): Tom and his dog went to the park. They saw a big tree. The tree was big and brown and shiny. They wanted to see the hill and the slide. They thought it would be fun. But the box were too high. The sun was shining. Tom and Lily looked at the tree. They saw a big dog in the forest. The dog wanted to play with the ball. "Can we play with it?" Tim asked. "Yes, I do. It's a toy," Tom said. "Maybe the dog is very good." "Maybe it was not a big, dangerous bird," the cat said. He started to cry.
Prompt: One day, a brave little knight wanted to explore the dark forest.
Greedy: One day, a brave little knight wanted to explore the dark forest. He was so excited to see the beautiful tree. He ran to the tree and saw a big tree. He was so excited to see the tree. He ran to the tree and saw a big tree. He was so happy and he ran to the tree. He was so happy to see the tree. He had a big, strong, red, and he was so happy. He had a big, strong, red, and he was so happy. He had a big, strong, red, and he was so happy. He had a big, red ball and a big, strong dog.
Sampled (T=0.8, top-k 50, top-p 0.95): One day, a brave little knight wanted to explore the dark forest. The little girl looked up and saw something strange. She was so happy! She went to the beach and ran outside. Inside the little girl noticed the little girl and the little girl ran out of the park. Suddenly, she heard a loud noise and started to shiver. The little girl saw the small girl and saw the little girl, who was worried that she wanted to help. She started to cry and the little girl ran back to her. She hugged the girl and said, "Don't be scared. I'm not so happy to show you." The little girl was disappointed and said, "
Prompt: Lily was sad because she lost her red ball. Then
Greedy: Lily was sad because she lost her red ball. Then she said to her mom, "I'm sorry, Mommy. I was just playing with you. I'm sorry I was mean to you." Her mom smiled and said, "I'm sorry, mom. I was just playing with you. I love you too." She said, "I love you too, mom. I love you too. I love you too."
Sampled (T=0.8, top-k 50, top-p 0.95): Lily was sad because she lost her red ball. Then she came closer and saw the big tree. It was shiny and made its mouth look silly. It looked scared and had a great day. The little boy was very happy and said hello to it, "Do you like to tell me how to play with it?" The little boy smiled and thanked the little boy for his snack. When he was finished, the little boy said, "That's a good idea! And the little boy! Let me help you." The little boy nodded and clapped his eyes. He was so happy and he hugged the little boy and said, "Thank you for helping
Limitations
- Trained only on synthetic TinyStories: simple children's-story English and narrow topics; it does not follow instructions or answer questions and does poorly outside this domain.
- Under-trained relative to the 20-tokens-per-parameter estimate; single-GPU, limited-time run.
- Packed training blocks let attention cross story boundaries; context is limited to 512 tokens.
- No instruction tuning, alignment or safety tuning; output can be repetitive, incoherent or inconsistent.
- Observed in my own samples:
- After training, my model generated readable sentences instead of the repeated unrelated words it produced before training. However, the stories were not consistently coherent.
- Greedy decoding often repeated the same phrases, such as "I will help you" and "The dog is angry."
- The model sometimes lost track of the main character. For example, the robot became a bird, and the knight's story switched to a little girl.
- Sampling produced more varied text, but it still included repetition, unrelated events, and grammar mistakes such as "the box were too high."
- Some descriptions did not make sense, such as "clapped his eyes." The model also struggled to continue the original problem, such as finding Lily's lost ball.
- These examples show that the model learned some common story patterns, but it still struggles with consistency and meaningful story development.
Usage
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer repo = "yahya-ai10/tinystories-75m-llama"tokenizer = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo).eval() prompt = "Once upon a time, a little robot found a book."inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False, return_token_type_ids=False)with torch.no_grad(): output = model.generate(**inputs, max_new_tokens=120, do_sample=True, temperature=0.8, top_k=50, top_p=0.95)print(tokenizer.decode(output[0], skip_special_tokens=True))
Notes: no BOS token is added (training used <eos> between stories only). Tested with transformers 4.46.3.
Credits
Dataset: Eldan & Li, TinyStories (2023). Planning rule: Hoffmann et al., Training Compute-Optimal Large Language Models (2022). Author: Yahya Majrashi.