bhautikv

mini-gpt-tinystories

Model Details

Table with columns: Parameter, Value
Parameter	Value
Architecture	GPT-2 (decoder-only transformer)
Parameters	17.7M
Layers	6
Hidden dimension	256
Attention heads	8
Vocabulary size	50,257 (GPT-2 BPE)
Context length	256 tokens
Tokenizer	GPT-2
Training data	TinyStories (2,119,719 stories)
Training framework	Hugging Face Transformers + Accelerate

Evaluation Results

Table with columns: Metric, Score, Interpretation
Metric	Score	Interpretation
Validation Loss	1.5902	Cross-entropy on held-out stories
Avg Loss (2K samples)	1.5700	Independent re-evaluation
Perplexity	4.81	Excellent — model confidently predicts next tokens
BLEU	0.1082	Reasonable for open-ended generation
Distinct-1	0.4108	Diverse vocabulary usage (>0.3 = good)
Distinct-2

A perplexity of 4.81 is excellent for TinyStories — the model has learned the simple vocabulary and story structure well. For comparison, random prediction would yield perplexity ~50,257 (vocab size).

Usage

Quick Start

python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "bhautikv/mini-gpt-tinystories"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16)
model.to("cuda" if torch.cuda.is_available() else "cpu")

prompt = "Once upon a time, a little girl named Lily"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

output = model.generate(
    **inputs,
    max_new_tokens=150,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.1,
    pad_token_id=tokenizer.eos_token_id,
)

print(tokenizer.decode(output, skip_special_tokens=True))

Batch Generation

python
prompts = [
    "Once upon a time, a little girl named Lily",
    "The brave knight approached the dragon and",
    "One day, a small bird found a shiny",
]

for prompt in prompts:
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    output = model.generate(
        **inputs,
        max_new_tokens=120,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
    )
    print(tokenizer.decode(output, skip_special_tokens=True))
    print("---")

Training Procedure

Training Hyperparameters

Table with columns: Hyperparameter, Value
Hyperparameter	Value
Learning rate	3e-4
Train batch size (per device)	32
Eval batch size (per device)	32
Gradient accumulation steps	2
Total effective batch size	128
Optimizer	AdamW (betas=(0.9, 0.999), eps=1e-8)
LR scheduler	Cosine
Warmup steps	1,000
Number of epochs

Training Results

Table with columns: Training Loss, Epoch, Step, Validation Loss
Training Loss	Epoch	Step	Validation Loss
3.1502	0.0695	1000	3.0154
2.4582	0.1390	2000	2.3299
2.2208	0.2084	3000	2.1053
2.1085	0.2779	4000	1.9927
2.0266

Training Infrastructure

Platform: Kaggle Notebook (GPU T4 x2)
GPUs: 2× NVIDIA T4 (16GB each)
Distributed strategy: Accelerate launch with DistributedDataParallel
Experiment tracking: Trackio (Hugging Face)
Training time: ~6 hours (including data preprocessing)

Data Preprocessing

Loaded 2,119,719 training stories and 21,990 validation stories from TinyStories
Tokenized using GPT-2 BPE tokenizer (vocab size 50,257)
Concatenated all tokens and grouped into fixed 256-token blocks
Used DataCollatorForLanguageModeling with dynamic padding for training

Intended Uses & Limitations

Intended Uses

Educational demonstration of small-scale language model training
Generating simple children's stories with coherent structure
Baseline for experimenting with GPT-2 architecture modifications
Learning resource for distributed training with Hugging Face Accelerate

Limitations

Only trained on TinyStories — limited to simple vocabulary and short narratives
Small model size (17.7M params) limits complexity of generated stories
Context length of 256 tokens restricts long-form generation
Not suitable for production use, real-world text generation, or handling complex topics
May produce repetitive or nonsensical text outside the TinyStories domain

Framework Versions

Transformers 5.14.1
Pytorch 2.10.0+cu128
Datasets 5.0.0
Tokenizers 0.22.2
Accelerate (latest)
Trackio (latest)

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

bhautikv

Model Tree

Base

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

Model Details

Table with columns: Parameter, Value
Parameter	Value
Architecture	GPT-2 (decoder-only transformer)
Parameters	17.7M
Layers	6
Hidden dimension	256
Attention heads	8
Vocabulary size	50,257 (GPT-2 BPE)
Context length	256 tokens
Tokenizer	GPT-2
Training data	TinyStories (2,119,719 stories)
Training framework	Hugging Face Transformers + Accelerate

Evaluation Results

Table with columns: Metric, Score, Interpretation
Metric	Score	Interpretation
Validation Loss	1.5902	Cross-entropy on held-out stories
Avg Loss (2K samples)	1.5700	Independent re-evaluation
Perplexity	4.81	Excellent — model confidently predicts next tokens
BLEU	0.1082	Reasonable for open-ended generation
Distinct-1	0.4108	Diverse vocabulary usage (>0.3 = good)
Distinct-2

A perplexity of 4.81 is excellent for TinyStories — the model has learned the simple vocabulary and story structure well. For comparison, random prediction would yield perplexity ~50,257 (vocab size).

Usage

Quick Start

python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "bhautikv/mini-gpt-tinystories"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16)
model.to("cuda" if torch.cuda.is_available() else "cpu")

prompt = "Once upon a time, a little girl named Lily"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

output = model.generate(
    **inputs,
    max_new_tokens=150,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.1,
    pad_token_id=tokenizer.eos_token_id,
)

print(tokenizer.decode(output, skip_special_tokens=True))

Batch Generation

python
prompts = [
    "Once upon a time, a little girl named Lily",
    "The brave knight approached the dragon and",
    "One day, a small bird found a shiny",
]

for prompt in prompts:
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    output = model.generate(
        **inputs,
        max_new_tokens=120,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
    )
    print(tokenizer.decode(output, skip_special_tokens=True))
    print("---")

Training Procedure

Training Hyperparameters

Table with columns: Hyperparameter, Value
Hyperparameter	Value
Learning rate	3e-4
Train batch size (per device)	32
Eval batch size (per device)	32
Gradient accumulation steps	2
Total effective batch size	128
Optimizer	AdamW (betas=(0.9, 0.999), eps=1e-8)
LR scheduler	Cosine
Warmup steps	1,000
Number of epochs

Training Results

Table with columns: Training Loss, Epoch, Step, Validation Loss
Training Loss	Epoch	Step	Validation Loss
3.1502	0.0695	1000	3.0154
2.4582	0.1390	2000	2.3299
2.2208	0.2084	3000	2.1053
2.1085	0.2779	4000	1.9927
2.0266

Training Infrastructure

Platform: Kaggle Notebook (GPU T4 x2)
GPUs: 2× NVIDIA T4 (16GB each)
Distributed strategy: Accelerate launch with DistributedDataParallel
Experiment tracking: Trackio (Hugging Face)
Training time: ~6 hours (including data preprocessing)

Data Preprocessing

Loaded 2,119,719 training stories and 21,990 validation stories from TinyStories
Tokenized using GPT-2 BPE tokenizer (vocab size 50,257)
Concatenated all tokens and grouped into fixed 256-token blocks
Used DataCollatorForLanguageModeling with dynamic padding for training

Intended Uses & Limitations

Intended Uses

Educational demonstration of small-scale language model training
Generating simple children's stories with coherent structure
Baseline for experimenting with GPT-2 architecture modifications
Learning resource for distributed training with Hugging Face Accelerate

Limitations

Only trained on TinyStories — limited to simple vocabulary and short narratives
Small model size (17.7M params) limits complexity of generated stories
Context length of 256 tokens restricts long-form generation
Not suitable for production use, real-world text generation, or handling complex topics
May produce repetitive or nonsensical text outside the TinyStories domain

Framework Versions

Transformers 5.14.1
Pytorch 2.10.0+cu128
Datasets 5.0.0
Tokenizers 0.22.2
Accelerate (latest)
Trackio (latest)

mini-gpt-tinystories

README

Model Details

Evaluation Results

Usage

Quick Start

Batch Generation

Training Procedure

Training Hyperparameters

Training Results

Training Infrastructure

Data Preprocessing

Intended Uses & Limitations

Intended Uses

Limitations

Framework Versions

Explore FriendliAI today

README

Model Details

Evaluation Results

Usage

Quick Start

Batch Generation

Training Procedure

Training Hyperparameters

Training Results

Training Infrastructure

Data Preprocessing

Intended Uses & Limitations

Intended Uses

Limitations

Framework Versions