GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
John-Machado
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
John-Machado
Model Tree
Input Modalities
Output Modalities
Supported Functionality
python
from transformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from_pretrained("John-Machado/gpt2-small-fineweb-edu")tokenizer = AutoTokenizer.from_pretrained("John-Machado/gpt2-small-fineweb-edu") ids = tokenizer("The water cycle is", return_tensors="pt").input_idsout = model.generate(ids, max_new_tokens=40, do_sample=True, top_k=50, pad_token_id=tokenizer.eos_token_id)print(tokenizer.decode(out[0]))Or load it into the readable model from the GitHub repo, where every weight has a plain name:
python
from gpt2.from_hf import load_gpt2model = load_gpt2("John-Machado/gpt2-small-fineweb-edu")Sampled with top_k=50 (seed 0); the prompt is in bold:
The water cycle is a fundamental system of our planet that is crucial for life on the planet. The Water Cycle, while essential for life in general, can't be changed. Water is essential for life on Earth
The French Revolution began in 1789, with the establishment of the "New State of the People" at Versailles. Napoleon III and Napoleon Bonaparte, a French Revolutionary party led by Giacomo Meyerbeer,
Fluent, on-topic, textbook-style English -- and, like any 124M model, confident inventions (1789 and Versailles are right; Napoleon III and the composer Meyerbeer are not).
Every step of this run -- preparing the data, a smoke test, training, and the comparison with OpenAI's model -- is written out in the repo: Train GPT-2 small from scratch.
| Architecture | GPT-2 small: 12 layers, 768 wide, 12 heads, context 1024, tied embeddings -- 124,439,808 parameters |
| Data | FineWeb-Edu sample-10BT (revision 87f09149ef4734204d70ed1d046ddc9ca3f2b8f9), GPT-2 tokenizer; 9,854,302,149 training tokens, read once, in order |
| Steps | 18,794 steps of 524,288 tokens (micro-batch 16 × 32 accumulation × 1024) |
| Optimizer | AdamW, betas (0.9, 0.95), weight decay 0.1 on the 2-D weights, gradient norm capped at 1.0 |
| Learning rate | linear warmup over 715 steps to 6e-4, then cosine down to 6e-5 |
| Dropout | 0.0 |
| Precision and speed | bf16 autocast, TF32, fused attention (FlashAttention via PyTorch SDPA), torch.compile |
| Hardware | one NVIDIA A100 40GB on Google Colab Pro+: 15.6 hours at ~176,000 tokens/s |
| Final loss | train ~3.0; validation 3.0113 on the in-training check (the first 327,680 held-out tokens) |
The learning rates and batch size follow the GPT-3 paper's settings for its 125M model.
The weights are released under the MIT license, like the code. The training data is FineWeb-Edu by Hugging Face (ODC-By 1.0; its use is also subject to Common Crawl's terms of use). The tokenizer is OpenAI's GPT-2 tokenizer. Architecture: OpenAI's GPT-2 (Radford et al., 2019).
from transformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from_pretrained("John-Machado/gpt2-small-fineweb-edu")tokenizer = AutoTokenizer.from_pretrained("John-Machado/gpt2-small-fineweb-edu") ids = tokenizer("The water cycle is", return_tensors="pt").input_idsout = model.generate(ids, max_new_tokens=40, do_sample=True, top_k=50, pad_token_id=tokenizer.eos_token_id)print(tokenizer.decode(out[0]))from gpt2.from_hf import load_gpt2model = load_gpt2("John-Machado/gpt2-small-fineweb-edu")