shreyash-pandey-katni

phoenix-350m-base

README

License: apache-2.0

Highlights

Chinchilla-optimal (~28.5 tokens/param) — trained to the compute-efficient point for its size.
Punches above its token weight on knowledge/reasoning: on ARC it matches or beats models trained on 30×–180× more data, thanks to FineWeb-Edu's dense educational content.
Final validation perplexity 15.6 (~0.86 bits/byte on its own held-out set).

Architecture

Table

Type	Decoder-only (Qwen3-compatible)
Params	350M
Layers	24
d_model	1024
Heads	16 query / 4 KV (GQA)
Head dim	64, with QK-norm
FFN	SwiGLU, d_ff 3456
Norm	RMSNorm (pre-norm)
Positional	RoPE (θ=10000)
Context	2048
Vocab	32,000 (custom byte-level BPE)
Embeddings	tied

Training

Data: FineWeb-Edu (sample-10BT), ~9.99B tokens, one epoch.
Optimizer: 8-bit AdamW (bitsandbytes), lr 6e-4, WSD schedule (2000 warmup, final-10% linear decay).
Precision: bf16, gradient checkpointing, global batch 524,288 tokens.
Hardware: 1× RTX 3080 Ti (12 GB), ~7 days.

Evaluation (0-shot, `lm-evaluation-harness`, acc_norm where conventional)

Table with columns: Model, Train tok, HellaSwag, PIQA, ARC-e, ARC-c, WinoGrande, OBQA, SciQ
Model	Train tok	HellaSwag	PIQA	ARC-e	ARC-c	WinoGrande	OBQA	SciQ
Phoenix-350M	10B	39.2	66.3	53.3	29.4	50.4	33.2	80.7
Pythia-410M	300B

Bits-per-byte (same-corpus, WikiText-103): Phoenix 0.98 vs Pythia-410M 0.86 vs Pythia-160M 1.19.

How to read this: Phoenix trained on 10B tokens — 30×–400× less data than the peers above. It's competitive-to-better on ARC (knowledge/reasoning, where FineWeb-Edu shines — it beats Apple's OpenELM-450M on both ARC-easy and ARC-challenge despite 180× less data), but its smaller token budget shows on HellaSwag, PIQA, WinoGrande and language-modeling metrics (BPB, LAMBADA), which scale with raw token count. MMLU is ~random at this scale, as expected for a 350M base model.

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("shreyash-pandey-katni/phoenix-350m-base")
model = AutoModelForCausalLM.from_pretrained("shreyash-pandey-katni/phoenix-350m-base")
ids = tok("The process of photosynthesis begins when", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=60)[0], skip_special_tokens=True))

ONNX weights are provided under onnx/ for optimum / onnxruntime / transformers.js.

Limitations

Base model — not instruction-tuned or aligned; it completes text, it doesn't follow chat instructions.
English only, ~10B tokens — limited world knowledge; will hallucinate and repeat.
No safety/RLHF filtering. Use for research and as a small-model baseline, not production.

License

Apache-2.0. Trained on FineWeb-Edu (ODC-BY).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

shreyash-pandey-katni

Model Tree

Base

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Highlights

Chinchilla-optimal (~28.5 tokens/param) — trained to the compute-efficient point for its size.
Punches above its token weight on knowledge/reasoning: on ARC it matches or beats models trained on 30×–180× more data, thanks to FineWeb-Edu's dense educational content.
Final validation perplexity 15.6 (~0.86 bits/byte on its own held-out set).

Architecture

Table

Type	Decoder-only (Qwen3-compatible)
Params	350M
Layers	24
d_model	1024
Heads	16 query / 4 KV (GQA)
Head dim	64, with QK-norm
FFN	SwiGLU, d_ff 3456
Norm	RMSNorm (pre-norm)
Positional	RoPE (θ=10000)
Context	2048
Vocab	32,000 (custom byte-level BPE)
Embeddings	tied

Training

Data: FineWeb-Edu (sample-10BT), ~9.99B tokens, one epoch.
Optimizer: 8-bit AdamW (bitsandbytes), lr 6e-4, WSD schedule (2000 warmup, final-10% linear decay).
Precision: bf16, gradient checkpointing, global batch 524,288 tokens.
Hardware: 1× RTX 3080 Ti (12 GB), ~7 days.

Evaluation (0-shot, `lm-evaluation-harness`, acc_norm where conventional)

Table with columns: Model, Train tok, HellaSwag, PIQA, ARC-e, ARC-c, WinoGrande, OBQA, SciQ
Model	Train tok	HellaSwag	PIQA	ARC-e	ARC-c	WinoGrande	OBQA	SciQ
Phoenix-350M	10B	39.2	66.3	53.3	29.4	50.4	33.2	80.7
Pythia-410M	300B

Bits-per-byte (same-corpus, WikiText-103): Phoenix 0.98 vs Pythia-410M 0.86 vs Pythia-160M 1.19.

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("shreyash-pandey-katni/phoenix-350m-base")
model = AutoModelForCausalLM.from_pretrained("shreyash-pandey-katni/phoenix-350m-base")
ids = tok("The process of photosynthesis begins when", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=60)[0], skip_special_tokens=True))

ONNX weights are provided under onnx/ for optimum / onnxruntime / transformers.js.

Limitations

Base model — not instruction-tuned or aligned; it completes text, it doesn't follow chat instructions.
English only, ~10B tokens — limited world knowledge; will hallucinate and repeat.
No safety/RLHF filtering. Use for research and as a small-model baseline, not production.

License

Apache-2.0. Trained on FineWeb-Edu (ODC-BY).

phoenix-350m-base

README

Highlights

Architecture

Training

Evaluation (0-shot, lm-evaluation-harness, acc_norm where conventional)

Usage

Limitations

License

Explore FriendliAI today

README

Highlights

Architecture

Training

Evaluation (0-shot, lm-evaluation-harness, acc_norm where conventional)

Usage

Limitations

License

Evaluation (0-shot, `lm-evaluation-harness`, acc_norm where conventional)

Evaluation (0-shot, `lm-evaluation-harness`, acc_norm where conventional)