ryze-ai

flash-archer-150M-2.0

README

License: apache-2.0

Model details

Table with columns: Field, Value
Field	Value
Architecture	Decoder-only transformer (Llama-family: pre-norm RMSNorm, RoPE, SwiGLU, tied embeddings)
Parameters	150.3M total (~138M non-embedding)
Layers	18
Hidden size	768
Attention heads	12 (head dim 64)
MLP hidden dim	2304 (SwiGLU)
Context length	1024 tokens
Vocabulary	16,000 (byte-level BPE, trained on 200k FineWeb-Edu docs)
Special tokens	`<pad>` `0` · `<unk>` `1` · `<bos>` `2` · `<eos>` `3`
RoPE θ	10,000
Training precision	fp16 with gradient scaling
Training tokens	~2.12B (step 16,200)
Training data	HuggingFaceFW/fineweb-edu `sample-10BT`
Hardware	1× Tesla T4 (15.6 GB VRAM), Kaggle
Training time	~7 hours
Final val loss	2.95 · val perplexity 19.2
Optimizer	AdamW (β₁=0.9, β₂=0.95, wd=0.1, cosine schedule, lr 3e-4 → 3e-5, 2% warmup)
Batch	micro-batch 1 × 1024 tokens × 128 grad-accum = ~131k tokens/step
License	Apache 2.0

Quickstart

python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "Norman89107/Flash-Archer-150M-2.0"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")
model.eval()

prompt = "Photosynthesis is the process by which"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(
    **inputs,
    max_new_tokens=120,
    temperature=0.8,
    top_k=50,
    top_p=0.95,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Sampling tips

Lower temperature (0.3–0.6) → more focused, factual output
Higher temperature (0.9–1.2) → more creative, more repetition
top_k=0 disables top-k; top_p=1.0 disables nucleus sampling
Context is 1024 tokens; longer prompts are cropped from the left

Training recipe

The full training code, tokenizer training, and streaming data pipeline are in the accompanying notebooks (archer-150m.ipynb for training, archer-150m-inference-colab.ipynb for inference). Key design choices:

Streaming, never materialized — FineWeb-Edu is streamed token-by-token; a fixed 2,000-doc holdout is reserved for validation and never seen by training.
Packed sequences — documents are concatenated with <eos> separators into fixed 1024-token chunks, so there is zero padding waste.
Activation checkpointing + micro-batch size 1 + gradient accumulation 128 — keeps the effective batch at ~131k tokens/step while fitting 1024-token training in 16 GB VRAM.
Persistent checkpoints — every 200 steps the model + optimizer + logs are mirrored to the HuggingFace Hub, so a crashed Kaggle session loses at most ~200 steps.

Intended use & limitations

Intended use: a compact, fast, English language model for experimentation, education, and as a baseline for small-model research. It writes coherent, on-topic prose and follows simple instructions.
Limitations: at 150M parameters it will hallucinate facts, repeat itself (especially at high temperature), and struggle with multi-step reasoning, math, and code. It has no safety alignment or RLHF — do not use it for applications without additional safeguards. Trained on web data; it may reflect biases and content present in FineWeb-Edu.

Conversion & reproducibility

This release was converted from the original training checkpoint (ckpt_step0016200.pt) into standard HuggingFace LlamaForCausalLM format. The only architectural difference is the RoPE convention (interleaved → half-split), handled by a fixed Q/K head-dimension permutation during conversion. Inference parity was verified: the converted model's logits match the original model's to within float32 rounding (max |diff| < 1e-5), and seeded generation produces identical output. See convert_to_hf.py and verify_parity.py in the project source.

Citation

If you use this model, please cite the training data and this repository:

markdown
@misc{flash-archer-150m-2,
  title  = {Flash-Archer-150M-2.0},
  author = {Norman89107},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/Norman89107/Flash-Archer-150M-2.0}},
  note   = {Decoder-only transformer trained from scratch on FineWeb-Edu (sample-10BT), ~2.1B tokens.}
}

markdown
@dataset{fineweb-edu,
  title  = {FineWeb-Edu},
  author = {Hugging Face},
  url    = {https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

ryze-ai

Model Tree

Base

this model

Input Modalities

Text

Output Modalities