beardymcgee

cascamini-llama-350m-edu-3b-v2

Model summary

Table with columns: Field, Value
Field	Value
Parameters	340,920,320
Architecture	Llama-style dense causal decoder
Layers	24
Hidden size	1,024
Attention heads	16
Key/value heads	4
MLP intermediate size	3,072
Context length	2,048 tokens
Tokenizer	GPT-2 BPE, vocabulary size 50,257
Positional encoding	RoPE, theta 10,000
Normalisation	RMSNorm
Activation	SwiGLU
Biases	Disabled
Embeddings	Input/output embeddings tied
Optimiser	AdamW
Training budget	3,000,041,472 tokens presented
Final fixed validation loss	2.9834
Final fixed validation perplexity	19.75

Training

The model was pretrained from scratch on a FineWeb-Edu-derived GPT-2 BPE token corpus.

The training run used:

a reproducible V2 trainer;
separate training and evaluation random-number streams;
fixed validation windows;
atomic checkpoint writes;
exact-resume state including model, optimiser, counters, and RNG state;
cosine learning-rate decay;
BF16 mixed-precision training.

The final run completed 45,777 optimiser updates. The final checkpoint was also the best checkpoint on the fixed validation set.

3,000,041,472 refers to tokens presented during training. Because training windows were sampled from the corpus, this is not a claim of three billion unique tokens.

Hugging Face export validation

The native CascaMini checkpoint was converted to LlamaForCausalLM format.

The export was validated by comparing native-model logits with Hugging Face Transformers logits before saving the BF16 weights. The parity test passed.

Usage

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "beardymcgee/cascamini-llama-350m-edu-adamw-3b-v2"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

prompt = "Photosynthesis is the process by which plants"

inputs = tokenizer(
    prompt,
    return_tensors="pt",
    add_special_tokens=False,
).to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=160,
        do_sample=True,
        temperature=0.65,
        top_k=40,
        top_p=0.90,
        repetition_penalty=1.08,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=False))

Recommended generation settings

These settings produced the most coherent outputs during manual testing:

text
do_sample=True
temperature=0.65
top_k=40
top_p=0.90
repetition_penalty=1.08
max_new_tokens=100-200

Greedy decoding is not recommended because the model frequently falls into repetition loops.

Example output

Prompt:

text
Photosynthesis is the process by which plants

Example continuation:

text
use light energy to make food. The process of photosynthesis is known as
photosynthesis. Why do plants need to be able to take in carbon dioxide from
the atmosphere? ...

This example demonstrates the model's learned educational style, but it should not be treated as factually reliable.

Intended use

Suitable uses include:

studying small causal language models;
architecture and optimiser comparisons;
tokenizer and data experiments;
reproducibility and exact-resume testing;
educational demonstrations of pretraining;
continued pretraining or supervised fine-tuning experiments.

Limitations

This model:

is not instruction tuned;
is not preference or safety tuned;
frequently produces plausible but incorrect factual statements;
performs poorly on mathematics and code generation;
can become repetitive, especially with greedy decoding;
may contradict itself during longer generations;
uses the GPT-2 tokenizer despite having a Llama-style architecture;
should not be used for medical, legal, financial, safety-critical, or other high-stakes decisions.

The model learned the form and tone of educational text more successfully than it learned reliable factual knowledge.

Evaluation notes

The fixed validation result is useful for comparisons made within the CascaMini project using the same tokenisation, data preparation, context length, and evaluation windows.

It should not be compared directly with unrelated models whose validation datasets or tokenisation differ.

Manual qualitative testing used prompts covering:

operating systems;
photosynthesis;
Roman history;
prime numbers;
Python factorial code.

The model showed improved fluency over earlier CascaMini checkpoints, while factual accuracy, mathematics, and code remained weak.

Files in this repository

Expected repository contents include:

text
config.json
generation_config.json
model.safetensors
tokenizer.json
tokenizer_config.json
export_report.json
README.md

Project status

This checkpoint closes the CascaMini GPT-2-tokenizer/FineWeb-Edu architecture phase.

The promoted direction for later work is:

text
Llama-style dense decoder
AdamW
reproducible V2 trainer
new tokenizer
broader data mixture
larger scaling experiments

Disclaimer

This is an experimental research artefact. Outputs may be incorrect, misleading, repetitive, or inappropriate. Users are responsible for evaluating generated content before relying on it.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.