1. Project Overview
This project implements and trains a 75-million-parameter Llama-style causal language model from scratch using the TinyStories dataset.
The objective is to explore the relationship between model size, training tokens, computational budget, and language modeling performance through scaling-law calculations and practical experimentation.
The model uses a decoder-only Transformer architecture with Grouped-Query Attention (GQA), Rotary Position Embeddings (RoPE), RMSNorm, and SwiGLU.
The initial experiment used 30,000 stories and 300 training steps. An improved experiment increased the dataset to 120,000 stories and extended training to 5,000 optimizer steps while preserving the same model architecture.
The improved model achieved:
- 74,998,272 trainable parameters
- 5,000 training steps
- 20.48 million processed training tokens
- Full Validation Loss: 2.1244
- Full Validation Perplexity: 8.3682
- Full Test Loss: 2.1353
- Full Test Perplexity: 8.4594
The model was initialized with random weights and trained from scratch. It was not fine-tuned from a pretrained language model.
2. Model Architecture
Table with columns: Component, Configuration| Component | Configuration |
|---|
| Architecture | Llama-style Causal Language Model |
| Model Type | Decoder-only Transformer |
| Trainable Parameters | 74,998,272 |
| Decoder Layers | 24 |
| Hidden Size | 512 |
| Intermediate Size | 1408 |
| Attention Heads | 8 |
| Key/Value Heads | 4 |
| Attention Type | Grouped-Query Attention (GQA) |
| Activation | SwiGLU |
| Positional Embeddings | RoPE |
| Normalization | RMSNorm |
| Vocabulary Size | 8192 |
| Training Context Length | 256 |
| Maximum Position Embeddings | 512 |
| Weight Tying | Enabled |
Architecture Rationale
The architecture was designed to approximately match a 75M-parameter target derived from the assignment's scaling-law calculations.
Grouped-Query Attention (GQA) was selected to reduce key/value parameters and KV-cache requirements compared with standard Multi-Head Attention (MHA).
The attention configurations were compared:
Table with columns: Attention Type, KV Heads, Parameters| Attention Type | KV Heads | Parameters |
|---|
| Multi-Query Attention (MQA) | 1 | 70,279,680 |
| Grouped-Query Attention (GQA) | 4 | 74,998,272 |
| Multi-Head Attention (MHA) | 8 | 81,289,728 |
GQA provides a balance between model capacity and computational efficiency.
The model retains 24 decoder layers, a hidden size of 512, and a training context length of 256 tokens.
3. Dataset and Preprocessing
Dataset: TinyStories
Language: English
Dataset License: CDLA-Sharing-1.0, as declared by the dataset provider.
TinyStories contains short synthetic children's stories designed for language modeling experiments.
Dataset Preparation
A total of 120,000 stories were sampled from the published training split.
The preprocessing pipeline included:
- Whitespace normalization.
- SHA-256 exact duplicate removal.
- Splitting complete stories into training, validation, and test sets.
- Training a byte-level BPE tokenizer using training stories only.
- Appending EOS tokens between stories.
- Packing tokens into non-overlapping sequences of 256 tokens.
- Discarding incomplete final token blocks.
Dataset Split
Table with columns: Split, Stories, Percentage| Split | Stories | Percentage |
|---|
| Training | 108,000 | 90% |
| Validation | 6,000 | 5% |
| Test | 6,000 | 5% |
| Total | 120,000 | 100% |
Random Seed: 42
Tokenized Training Data
- Packed training blocks: 89,770
- Packed training tokens: 22,981,120
- Vocabulary size: 8,192
- Context length: 256 tokens
Validation and test data were kept separate from gradient-based training.
4. Scaling Laws and Compute Budget
This project applies a simplified scaling-law planning approximation:
Compute approximation:
C ≈ 6ND
Where:
- C = training compute in FLOPs
- N = number of model parameters
- D = number of training tokens
Using the planning assumption:
D = 20N
The assignment specifies a compute budget of:
C = 6.75 × 10^17 FLOPs
Therefore:
N = sqrt(C / 120)
N = 75,000,000 parameters
D = 20 × 75,000,000
D = 1,500,000,000 tokens
Theoretical Targets
Table with columns: Metric, Value| Metric | Value |
|---|
| Compute Budget | 6.75e17 FLOPs |
| Target Parameters | 75,000,000 |
| Target Training Tokens | 1,500,000,000 |
| Actual Model Parameters | 74,998,272 |
| Updated Token Target | 1,499,965,440 |
These values represent theoretical planning targets, not the actual training budget used in the experiment.
Actual Improved Training Run
Table with columns: Metric, Value| Metric | Value |
|---|
| Training Steps | 5,000 |
| Input Tokens per Step | 4,096 |
| Total Input Tokens | 20,480,000 |
| Scored Next-Token Targets | 20,400,000 |
| Approximate Training Compute | 9.216e15 FLOPs |
| Coverage of Theoretical Token Target | 1.3654% |
The improved experiment increased training compute while keeping the model architecture and parameter count fixed.
This demonstrates the effect of increasing the training budget, although a single pair of experiments is insufficient to establish a general scaling law.
5. Training Configuration
The model was trained using PyTorch and Hugging Face Transformers on Google Colab.
Hardware
- GPU: NVIDIA Tesla T4
- Framework: PyTorch
- Model Library: Hugging Face Transformers 4.46.3
Hyperparameters
Table with columns: Hyperparameter, Value| Hyperparameter | Value |
|---|
| Optimizer | AdamW |
| Beta 1 | 0.9 |
| Beta 2 | 0.95 |
| Weight Decay | 0.1 |
| Learning Rate | 3e-4 |
| Learning Rate Scheduler | Cosine Decay |
| Warmup Steps | 100 |
| Microbatch Size | 4 |
| Gradient Accumulation Steps | 4 |
| Effective Batch Size | 16 |
| Context Length | 256 |
| Maximum Training Steps | 5,000 |
| Evaluation Interval | 250 steps |
| Evaluation Batches | 20 |
Training Statistics
Table with columns: Metric, Result| Metric | Result |
|---|
| Successful Optimizer Steps | 5,000 |
| Processed Input Tokens | 20,480,000 |
| Scored Training Targets | 20,400,000 |
| Training Duration | 118.21 minutes |
| Training Speed | Approximately 3,092 tokens/s |
| Peak Allocated GPU Memory | 1.48 GiB |
Training duration includes periodic evaluation and checkpoint saving within the training loop.
6. Model Evaluation
The model was evaluated using next-token cross-entropy loss and perplexity.
Perplexity is calculated as:
Perplexity = exp(Cross-Entropy Loss)
Lower loss and perplexity indicate better next-token prediction performance on the evaluated data.
Final Evaluation Results
Table with columns: Evaluation Split, Loss, Perplexity| Evaluation Split | Loss | Perplexity |
|---|
| Full Validation | 2.1244 | 8.3682 |
| Full Test | 2.1353 | 8.4594 |
The final evaluation used complete held-out validation and test splits.
Training Progress
The fixed validation subset was evaluated every 250 optimizer steps.
Table with columns: Training Step, Validation Loss, Perplexity| Training Step | Validation Loss | Perplexity |
|---|
| 250 | 3.8781 | 48.33 |
| 500 | 3.2397 | 25.53 |
| 750 | 2.9694 | 19.48 |
| 1,000 | 2.7918 | 16.31 |
| 1,500 | 2.5594 | 12.93 |
| 2,000 | 2.4250 | 11.30 |
| 2,500 | 2.3025 | 10.00 |
| 3,000 | 2.2237 | 9.24 |
| 5,000 | 2.0865 | 8.06 |
The fixed-subset evaluation differs from the final evaluation performed over the complete validation split.
Training and Validation Loss

The training and validation losses decreased substantially throughout the experiment.
The improvement became more gradual during later training steps, with no clear evidence of severe overfitting in the observed validation curve.
7. Baseline vs. Improved Model
Two experiments were conducted using the same 75M-parameter architecture.
Experimental Configurations
Table with columns: Setting, Baseline, Improved V2| Setting | Baseline | Improved V2 |
|---|
| Dataset Size | 30,000 stories | 120,000 stories |
| Training Steps | 300 | 5,000 |
| Training Tokens | 1.23M | 20.48M |
| Model Parameters | 74.998M | 74.998M |
| Context Length | 256 | 256 |
| Attention | GQA | GQA |
| Learning Rate | 3e-4 | 3e-4 |
Observed Results
Table with columns: Metric, Baseline, Improved V2| Metric | Baseline | Improved V2 |
|---|
| Full Validation Loss | 4.3317 | 2.1244 |
| Full Validation Perplexity | 76.07 | 8.37 |
| Full Test Loss | 4.3383 | 2.1353 |
| Full Test Perplexity | 76.57 | 8.46 |
The improved experiment produced substantially lower evaluation losses and perplexities.
Important comparison limitation: The dataset size, data splits, and tokenizer training corpus changed between experiments. Therefore, the numerical differences are descriptive and should not be interpreted as a controlled, directly comparable perplexity improvement.
A strictly controlled comparison would require using the same tokenizer and identical held-out evaluation data.
Key Observation
Increasing the amount of training data and the number of optimizer steps was associated with substantially better language modeling performance without modifying the model's architecture.
8. Text Generation Examples
Prompt
Once upon a time, a little robot found a book.
Before Training
Once upon a time, a little robot found a book.."."."."."."."."."."."."."."."."."."."."."."."."."."."."."."."."."."."."."."."ardenardenardenardenardenardenardenardenardenardenardenardenardenarden
After Training
Once upon a time, a little robot found a book. The robot was very happy and said, "Thank you, little robot! You are so kind!" The robot and the robot became good friends. They played together every day. The robot was very happy and never wanted to leave. The robot and the robot became best friends. They would play together every
Observations
Before training, the randomly initialized model produced repetitive and largely meaningless sequences.
After training, the model demonstrated:
- More grammatically coherent sentences.
- Improved next-token prediction.
- Better retention of the story topic.
- More recognizable children's story structure.
However, repetitive phrases and limited narrative development remain noticeable.
9. How to Use the Model
Installation
pip install transformers==4.46.3 torch safetensors
Load the Model
from transformers import AutoTokenizer, AutoModelForCausalLMimport torch model_id = ( "jawaher-leen-ai/" "tinystories-llama-75m-from-scratch") tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id) model.eval() prompt = "Once upon a time, a little robot found a book." inputs = tokenizer( prompt, return_tensors="pt") with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=60, do_sample=False, pad_token_id=tokenizer.pad_token_id, eos_token_id=tokenizer.eos_token_id ) generated_text = tokenizer.decode( outputs[0], skip_special_tokens=True) print(generated_text)
Optional: Sample More Diverse Stories
For more varied output, sampling can be enabled:
with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=160, do_sample=True, temperature=0.8, top_p=0.9, repetition_penalty=1.1, pad_token_id=tokenizer.pad_token_id, eos_token_id=tokenizer.eos_token_id ) print( tokenizer.decode( outputs[0], skip_special_tokens=True ))
Sampling may produce more diverse text, but does not improve the model's learned predictive ability.
10. Limitations
This is a small research and educational language model trained on a narrow English storytelling dataset.
Known limitations include:
- Repetition of words, sentences, and story patterns.
- Limited long-range narrative coherence.
- Possible grammatical errors.
- Restricted general knowledge.
- Potentially inaccurate or fabricated statements.
- Limited performance outside children's story generation.
- A relatively small training token budget compared with the theoretical scaling-law target.
The model is not instruction-tuned and is not intended to function as a general-purpose assistant.
It is also not intended for high-stakes or production decision-making.
11. Conclusion
This project demonstrates the complete process of designing, initializing, training, evaluating, and documenting a small Llama-style language model from scratch.
The experiment connects scaling-law calculations with practical model architecture selection and training.
By increasing the dataset from 30,000 to 120,000 stories and extending training from 300 to 5,000 steps, the improved experiment achieved substantially lower evaluation loss and perplexity.
The model architecture remained fixed at approximately 75M parameters, using Grouped-Query Attention.
The results illustrate how additional training data and compute can improve a small language model, while also highlighting the importance of controlled evaluation when comparing experiments.
12. Credits and References
Project Authors:
Dataset:
Research:
Frameworks:
- PyTorch
- Hugging Face Transformers
- Hugging Face Datasets
- Hugging Face Tokenizers
License
The model is released under the Apache License 2.0.
The TinyStories dataset is distributed separately under the license declared by its provider (CDLA-Sharing-1.0). Users should review the dataset's applicable terms independently.