1. Architecture
Table with columns: Item, Value| Item | Value |
|---|
| Family | Llama-style decoder (RMSNorm, RoPE, SwiGLU, grouped-query attention) |
| Total parameters | 74,156,160 |
| Layers | 14 |
| Hidden size | 640 |
| Attention (query) heads | 10 (head dim 64) |
| Key/value heads | 5 (GQA, 2 query heads share each KV head) |
| Intermediate (MLP) size | 1728 |
| Vocabulary size | 16,384 (byte-level BPE trained on the training split only) |
| Context length | 256 tokens during training (config max_position_embeddings = 512) |
| Embeddings | tied input/output embeddings; no attention or MLP biases |
Parameter breakdown: {'embedding (shared with output)': 10485760, 'attention': 17203200, 'SwiGLU': 46448640, 'normalization / other': 18560}.
Why these choices
- Budget first. With C = 6.75e+17 FLOPs, C = 6ND and D = 20N give C = 120N^2, so N = sqrt(C/120) = 75,000,000 and D = 1,500,000,000. This puts the model inside the required 60-100M range.
- Small vocabulary (16,384). Web text has more distinct words than children's stories, so the vocabulary is larger than the lab default (8,192) but still small. A GPT-2-sized vocabulary (50,257) would spend about 32.2M parameters on embeddings; ours uses 10.49M (14.1% of the model), leaving the rest for the transformer layers. Tying input and output embeddings avoids a second copy.
- Depth/width. Head dimension is fixed at 64 (efficient with PyTorch SDPA kernels) and H = 640 gives 10 heads (wider and shallower than the lab reference H = 512, L = 24). The layer count (14) is solved from the budget with the exact parameter formula, so the actual size is within 1.125% of the target.
- GQA (10 query / 5 KV heads). Halves the KV projections and the KV cache compared with full multi-head attention, at a small parameter saving versus MHA (5,734,400 fewer parameters than MHA for this shape).
- SwiGLU with I = 1728. About 8/3 x H rounded up to a multiple of 64, the standard size that keeps the MLP parameter count comparable to a 4x GELU MLP.
- Context 256. A short context keeps attention cost and memory low enough for a single Colab GPU.
2. Dataset
- Source:
HuggingFaceFW/fineweb-edu subset sample-10BT (Penedo et al., 2024, arXiv:2406.17557), streamed from the training split.
- Language: English (educational web pages filtered from Common Crawl).
- Licence: ODC-By 1.0 (Open Data Commons Attribution License).
- Preprocessing: whitespace normalisation, exact de-duplication (SHA-256 of normalised text), then a shuffled sample of 25,000 unique documents. Documents are split before tokenisation and packing so no document is shared between splits; the tokenizer is trained on the training split only. Documents are joined with an
<eos> token and cut into non-overlapping blocks of 256 tokens (the final incomplete block of each split is dropped).
- Splits (documents): train 22,500 (90%), validation 1,250 (5%), test 1,250 (5%), seed 42.
- Splits (packed tokens): train 23,852,270, validation 1,295,574, test 1,236,267.
3. Scaling-law calculations
Table with columns: Quantity, Value| Quantity | Value |
|---|
| Compute budget C | 6.75e+17 FLOPs |
| Planned parameters N (from C = 6ND, D = 20N) | 75,000,000 |
| Planned training tokens D = 20N | 1,500,000,000 |
| Actual parameters after rounding layers | 74,156,160 |
| Updated full token target 20 x actual N | 1,483,123,200 |
| Actual input tokens processed in training | 8,192,000 |
| Scored next-token targets | 8,160,000 |
| Coverage of the full token target | 0.552% |
| Approximate compute actually used (6 x N x tokens) | 3.645e+15 FLOPs |
The run is intentionally far below the compute-optimal budget (it covers 0.55% of it) because the full budget would take about 362,091 optimizer updates, which does not fit a free Colab session. The model is therefore under-trained relative to the scaling-law target.
4. Training setup
Table with columns: Item, Value| Item | Value |
|---|
| Hardware | Tesla T4 (peak allocated memory 1.55 GiB) |
| Precision | mixed precision (bfloat16), FP32 master weights, gradient checkpointing |
| Optimizer | AdamW, betas (0.9, 0.95), weight decay 0.1, gradient clipping 1.0 |
| Learning rate | 0.0003 peak, 100 warm-up steps, cosine decay |
| Micro-batch / accumulation | 4 sequences x 4 steps |
| Effective batch size | 16 sequences = 4,096 tokens per update |
| Successful optimizer updates | 2000 |
| Training duration | 41.3 minutes (3,304 tokens/s), excluding evaluation |
| Seed | 42 |
5. Evaluation
Next-token cross-entropy (nats per token) and perplexity = exp(loss), computed with the model's own tokenizer.
Table with columns: Evaluation, Loss, Perplexity, Scored tokens| Evaluation | Loss | Perplexity | Scored tokens |
|---|
| Before training (fixed validation subset) | 9.839 | 18758.5 | 20,400 |
| Full validation split (after training) | 5.650 | 284.38 | 1,290,300 |
| Full test split (held out, evaluated once at the end) | 5.659 | 286.83 | 1,231,395 |
Perplexity is only comparable between models using the same tokenizer, dataset and evaluation procedure.

The training curve shows the loss of the last effective batch at each evaluation point; the validation curve uses the same fixed subset of the validation split throughout training.
7. Example outputs
Prompt: The Roman Empire was one of the largest
Greedy decoding:
The Roman Empire was one of the largest and most important and more than the most important and more than the most important. The most important and more of the most important groups of the world is the most important and more important to the world. The most important and more of the most important things is to be the most important and more of the most important things. The most important is that the world is not a good way to be the same
Sampling (temperature 0.8, top-p 0.95, repetition penalty 1.1):
The Roman Empire was one of the largest part of the K. H.D. Y.E. In 1828, the two years later had not been the great of the world but the war is now considered in the first time. The Santon, he became the main place of these, and the first, were the most smallest. In this study, the S.S. has, who are the same in the
Prompt: Students learn best when teachers
Greedy decoding:
Students learn best when teachers are not able to be able to be able to understand their own. The best way to be able to be a good way to be able to be a good way to do. The best way to be a good way to be a good way to be able to the right. The best way to be a good way to be a good way to do. The best way to be a good way
Sampling (temperature 0.8, top-p 0.95, repetition penalty 1.1):
Students learn best when teachers have the same way to take on the world. And how do I think about their students? What is I found in the history of the world? How does I want the information in the situation? What we see that it is not not being in the same of the idea and then we can't have a right and a person who has been taken out and will be a great and effective life.
Prompt: Water boils at 100 degrees Celsius because
Greedy decoding:
Water boils at 100 degrees Celsius because of the world, and the world's most important and more than the world. The first time, the world’s most important and more than the world’s most important and more than the world. The first time, the world’s most important and more than the world’s life, the world’s life is to be the same. The world’s most important and the world
Sampling (temperature 0.8, top-p 0.95, repetition penalty 1.1):
Water boils at 100 degrees Celsius because they were not allowed to understand their knowledge. The following is one of the most important features on the world, where he or the U.S., we can’t do. They also are not getting better and they must want in a number of parts and we have to live in the early years and now all you can use as well as you are trying to have and keep the information more than
Prompt: The human heart has four chambers that
Greedy decoding:
The human heart has four chambers that are the most common in the world. The most common and more than the most important of the world is the most important part of the world. The most important and more people are the most important to the world. The most important is that the world is not a good way to be the same. The most important is that the world is not a good way to be the same. The most important
Sampling (temperature 0.8, top-p 0.95, repetition penalty 1.1):
The human heart has four chambers that have been done before the whole of our lives and by our own activities, you should not be aware of your life. I think to get them to look at all, we use the work to improve the environment you can. That's why we want to share the way to show yourself as we do this to know how it is being able to do. For example, we’ve developed a good
Text generated by the untrained model for the first prompt (greedy):
Photosynthesis is the process by which node node node node node node node node node node node node node node node node node node node node downt downt downt downt downt downt downt downt downt downt downt downt downt downt downt downt downt downt downt downt downt downt downt downt downt downt downt downt downt downt downt downt smile smile smile smile smile smile smile smile
8. Limitations
- Under-trained: only 0.55% of the scaling-law token budget was used, so text can be repetitive, drift off topic, or state false facts.
- Limited knowledge: trained on a small sample of English web text; it will not follow instructions, answer questions, or know facts, and it is not suitable for other domains or languages.
- Short context: trained on 256-token blocks; quality beyond that length is not tested even though the config allows 512 positions.
- No document masking: documents are packed together, so attention can cross document boundaries inside a block.
- No safety tuning: the model was not aligned or filtered beyond the dataset itself; web text can contain biased or outdated content.
- Evaluation scope: one run, one seed, one tokenizer; no comparison against other architectures.
9. Usage
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer repo = "waad12I/fineweb-edu-74m-llama"tokenizer = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo)model.eval() prompt = "The Roman Empire was one of the largest"inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False, return_token_type_ids=False)with torch.no_grad(): out = model.generate(**inputs, max_new_tokens=80, do_sample=True, temperature=0.8, top_p=0.95, repetition_penalty=1.1, pad_token_id=tokenizer.pad_token_id, eos_token_id=tokenizer.eos_token_id)print(tokenizer.decode(out[0], skip_special_tokens=True))
Citation / references
- Hoffmann et al., Training Compute-Optimal Large Language Models (2022), arXiv:2203.15556.
- Penedo et al., The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale (2024), arXiv:2406.17557.