✨ Highlights
- 🧠 Architecture: LLaMA-style (RMSNorm, RoPE, SwiGLU, Grouped-Query Attention)
- 🔢 Parameters: ~19.8M
- 📚 Training tokens: 400,000,000 (80 shards x ~5M tokens)
- 🧩 Tokenizer: custom Byte-Level BPE, 24k vocab (rust
tokenizers)
- 🧱 Training block size: 256 tokens
- 💾 Checkpointing: pushed to the Hub after every shard, resumable via
state.json
📊 Model Configuration
{
"model_type": "llama",
"hidden_size": 384,
"intermediate_size": 768,
"num_hidden_layers": 8,
"num_attention_heads": 6,
"num_key_value_heads": 3,
"vocab_size": 24000,
"tie_word_embeddings": true,
"hidden_act": "silu"
}
⚡ Quick Usage
from transformers import AutoTokenizer, AutoModelForCausalLM
repo = "SENAGI/Lily-2.0-20M"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)
prompt = "The meaning of life is"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
**inputs,
max_new_tokens=64,
do_sample=True,
temperature=0.7,
top_p=0.9,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
🗂️ Dataset
Trained on SENAGI/fineweb-edu-400M-raw, a 400M-token educational corpus split into 80 raw text
shards (part_01.txt ... part_80.txt).
🛠️ Training Details
- Optimizer: AdamW (beta1=0.9, beta2=0.95, weight_decay=0.1)
- Schedule: linear warm-up (10,000,000 tokens) + cosine decay
- Peak learning rate: 0.002
- Mixed precision: FP16 on GPU
- Gradient clipping: 1.0
- Checkpoint cadence: every shard, fully resumable
📜 License
MIT
Made with 🌸 by SENAGI