yorukot

gpt2-traning

Deploy Dedicated

README

License: apache-2.0

Model

Stock Hugging Face GPT2LMHeadModel: 12 layers, 12 attention heads, hidden width 768, context length 1,024, vocabulary 50,257, tied input/output embeddings.
124,439,808 unique parameters, saved in FP32.
GPT-2 tokenizer. No pretrained model weights were used.

Training

Data: C4 en train, tokenized with the GPT-2 tokenizer without BOS/EOS. Documents were truncated to 1,024 tokens and best-fit packed into 1,024-token rows, with positions restarting at 0 for each document.
Attention was restricted to each document during training (varlen flash attention). This is a training-time kernel only; the saved model is plain GPT-2.
Exact deduplication. The first 100,000 documents of validation.shuffle(seed=42) (the lab's test set) were excluded.
AdamW (fused): learning rate 1.8e-3, betas (0.9, 0.95), eps 1e-8, weight decay 0.1 on matrices only, gradient clipping 1.0, dropout 0, random seed 42.
Schedule: 150 warmup steps, full learning rate for the first half of the training time, then linear decay to 0.
Batch: 256 rows × 1,024 tokens = 262,144 token slots per step (including padding). 6,010 steps, about 1.58B token slots.
The vocabulary was padded to 50,304 during training and sliced back to 50,257 before saving.
Two NVIDIA H200 GPUs, BF16 autocast, torch.compile, 29 minutes of training. PyTorch 2.14.0, Transformers 5.17.0.

Evaluation

The evaluation set is the 10,000 C4 en validation documents that follow the lab's 100,000 test documents in validation.shuffle(seed=42). It is scored like the lab's judge:

Each document is scored on its own, from position 0, with no BOS.
Each document is truncated to 1,024 tokens.
The score is total loss divided by the number of scored tokens.

Table with columns: Split, Documents, Loss, Perplexity
Split	Documents	Loss	Perplexity
Local validation	10,000	3.3149	27.52

Load the checkpoint

python

from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "yorukot/gpt2-traning"tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id)

This is a base language model for an educational experiment, without instruction tuning. Its web training data can produce inaccurate or biased text.

Training run on W&B.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

yorukot

Model Tree

Base

this model

Input Modalities

Text

Output Modalities