tegancao

owt-45m

How to use

python

import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer tok = AutoTokenizer.from_pretrained("tegancao/owt-45m")model = AutoModelForCausalLM.from_pretrained("tegancao/owt-45m") torch.manual_seed(0)ids = tok("<|endoftext|>" + "The new study shows", return_tensors="pt").input_ids   # start like a new documentout = model.generate(ids, max_new_tokens=100, do_sample=True, temperature=0.8, top_p=0.9)print(tok.decode(out[0, 1:], skip_special_tokens=True))

Sample output (the code above, as is):

text

The new study shows that people with mental health disorders are more likely to be treated with mental health conditions. The study, published in the journal PLoS ONE in New York, found that just 3% of people with mental health issues, compared with 7% of people with mental health, had symptoms of alcoholism, which is nearly the same as those with mental health problems. The study showed that mental health issues were more likely to be related to physical health than mental health problems. The

It is a base model: it continues text and does not follow instructions or answer questions. Prompts that look like the start of its training documents work best.

Training

Table

Training data	a sample of OpenWebText (web pages linked from Reddit; a 2.7B-token sample)
Tokenizer	byte-level BPE trained on the same data, 32,000-token vocabulary, GPT-2 pre-tokenization pattern; `<
Architecture	Llama-style decoder: 4 layers, d_model 512, 16 heads (head dim 32), SwiGLU FFN (d_ff 1344), RMSNorm (pre-norm), RoPE (θ = 10,000), no biases, untied input/output embeddings
Context length	256 tokens
Optimizer	AdamW (β₁ 0.9, β₂ 0.95, ε 1e-8, weight decay 0.1), gradient clipping at 1.0
Learning rate	3e-3 peak, 1,000 linear warm-up steps, cosine decay to 3e-4
Batch	32 sequences × 256 tokens = 8,192 tokens per step
Steps / tokens	40,000 steps = 327.68M tokens
Precision / hardware	float32, one NVIDIA L4 GPU
Training time	~4.5 hours

The model was implemented and trained with my own from-scratch PyTorch code, then converted to the standard LlamaForCausalLM format (RoPE pairs reordered in the Q/K projections). The conversion was verified: the converted model's logits match the original within ~2 × 10⁻⁵, greedy generations are identical, and the tokenizer produces identical token ids on 2 MB of validation text.

Evaluation

Table with columns: Metric, Value
Metric	Value
Validation loss (cross-entropy, nats per token)	4.154
Validation perplexity	63.7

Measured on held-out validation text at the end of training, averaged over 20 random batches (about 164K tokens). The model saw about 12% of its training data (one partial pass over a 2.7B-token sample).

Intended use

Learning how language models work: inspecting a small model, experimenting with sampling (temperature, top-p), seeing how training data shapes behavior.
Fast experiments that need a tiny model: decoding methods, interpretability, testing inference code.
Demos of what a model trained from scratch on a small budget can and cannot do.

Out of scope

Any factual use: the model makes things up.
Anything user-facing or in production: there is no instruction tuning and no safety tuning.
Decisions about people, or any application where mistakes matter.

Limitations

Continues text instead of answering it (base model, no instruction tuning).
Loose logic and weak tracking of who is who over longer passages; falls into repetition loops, especially with greedy decoding.
Sees at most the last 256 tokens.
Generates convincing-looking but false text: invented quotes attributed to real people and organizations, made-up studies and statistics (like the "PLoS ONE" study in the sample above). Never treat its output as fact.
Trained on unfiltered web text, so it may produce biased or offensive content.
Much of its parameter count is vocabulary: the 32K-token embedding and output layers hold 32.8M of the 45.2M parameters; the 4 Transformer blocks are the same 12.5M as in the TinyStories model.

Acknowledgments

The architecture and training setup follow the publicly available materials of Stanford CS336 (Language Modeling from Scratch). Training data: a sample of OpenWebText (web pages linked from Reddit; a 2.7B-token sample). Please see the dataset's own card for its license and terms.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

tegancao

Model Tree

Base

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

How to use

python

import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer tok = AutoTokenizer.from_pretrained("tegancao/owt-45m")model = AutoModelForCausalLM.from_pretrained("tegancao/owt-45m") torch.manual_seed(0)ids = tok("<|endoftext|>" + "The new study shows", return_tensors="pt").input_ids   # start like a new documentout = model.generate(ids, max_new_tokens=100, do_sample=True, temperature=0.8, top_p=0.9)print(tok.decode(out[0, 1:], skip_special_tokens=True))

import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer tok = AutoTokenizer.from_pretrained("tegancao/owt-45m")model = AutoModelForCausalLM.from_pretrained("tegancao/owt-45m") torch.manual_seed(0)ids = tok("<|endoftext|>" + "The new study shows", return_tensors="pt").input_ids   # start like a new documentout = model.generate(ids, max_new_tokens=100, do_sample=True, temperature=0.8, top_p=0.9)print(tok.decode(out[0, 1:], skip_special_tokens=True))

Sample output (the code above, as is):

text

The new study shows that people with mental health disorders are more likely to be treated with mental health conditions.The study, published in the journal PLoS ONE in New York, found that just 3% of people with mental health issues, compared with 7% of people with mental health, had symptoms of alcoholism, which is nearly the same as those with mental health problems.The study showed that mental health issues were more likely to be related to physical health than mental health problems.The

The new study shows that people with mental health disorders are more likely to be treated with mental health conditions. The study, published in the journal PLoS ONE in New York, found that just 3% of people with mental health issues, compared with 7% of people with mental health, had symptoms of alcoholism, which is nearly the same as those with mental health problems. The study showed that mental health issues were more likely to be related to physical health than mental health problems. The

It is a base model: it continues text and does not follow instructions or answer questions. Prompts that look like the start of its training documents work best.

Training

Table

Training data	a sample of OpenWebText (web pages linked from Reddit; a 2.7B-token sample)
Tokenizer	byte-level BPE trained on the same data, 32,000-token vocabulary, GPT-2 pre-tokenization pattern; `<
Architecture	Llama-style decoder: 4 layers, d_model 512, 16 heads (head dim 32), SwiGLU FFN (d_ff 1344), RMSNorm (pre-norm), RoPE (θ = 10,000), no biases, untied input/output embeddings
Context length	256 tokens
Optimizer	AdamW (β₁ 0.9, β₂ 0.95, ε 1e-8, weight decay 0.1), gradient clipping at 1.0
Learning rate	3e-3 peak, 1,000 linear warm-up steps, cosine decay to 3e-4
Batch	32 sequences × 256 tokens = 8,192 tokens per step
Steps / tokens	40,000 steps = 327.68M tokens
Precision / hardware	float32, one NVIDIA L4 GPU
Training time	~4.5 hours

Evaluation

Table with columns: Metric, Value
Metric	Value
Validation loss (cross-entropy, nats per token)	4.154
Validation perplexity	63.7

Intended use

Learning how language models work: inspecting a small model, experimenting with sampling (temperature, top-p), seeing how training data shapes behavior.
Fast experiments that need a tiny model: decoding methods, interpretability, testing inference code.
Demos of what a model trained from scratch on a small budget can and cannot do.

Out of scope

Any factual use: the model makes things up.
Anything user-facing or in production: there is no instruction tuning and no safety tuning.
Decisions about people, or any application where mistakes matter.

Limitations

Continues text instead of answering it (base model, no instruction tuning).
Loose logic and weak tracking of who is who over longer passages; falls into repetition loops, especially with greedy decoding.
Sees at most the last 256 tokens.
Generates convincing-looking but false text: invented quotes attributed to real people and organizations, made-up studies and statistics (like the "PLoS ONE" study in the sample above). Never treat its output as fact.
Trained on unfiltered web text, so it may produce biased or offensive content.
Much of its parameter count is vocabulary: the 32K-token embedding and output layers hold 32.8M of the 45.2M parameters; the 4 Transformer blocks are the same 12.5M as in the TinyStories model.

owt-45m

README

How to use

Training

Evaluation

Intended use

Out of scope

Limitations

Acknowledgments

Explore FriendliAI today

README

How to use

Training

Evaluation

Intended use

Out of scope

Limitations

Acknowledgments