Model Details
Architecture
Table with columns: Parameter, Value| Parameter | Value |
|---|
| Architecture | LLaMA (decoder-only) |
| Parameters | 22,026,624 |
| Hidden size | 384 |
| Intermediate size | 1,536 |
| Num layers | 8 |
| Num attention heads | 12 |
| Num KV heads | 12 |
| Max sequence length | 1,024 |
| Vocabulary size | 8,192 |
| Position encoding | RoPE (θ=10000.0) |
| Activation | SwiGLU |
| Normalization | Pre-RMSNorm |
| Weight tying | Yes (embedding ↔ lm_head) |
| Precision | fp32 |
Training Data
Table with columns: Source, Type, Size| Source | Type | Size |
|---|
| HumanEval | Code problems | ~164 examples |
| LAMBADA | Narrative text | ~5,000 examples |
Training Hyperparameters
Table with columns: Parameter, Value| Parameter | Value |
|---|
| Hardware | 15GB VRAM |
| Optimizer | AdamW |
| Learning rate | 1e-4 |
| Batch size | 128 |
| Epochs | 10 |
| Training time | ~2 minutes |
Tokenizer
- Type: BPE (Byte-Level)
- Vocabulary: 8,192 tokens
- Pre-tokenizer: ByteLevel (no prefix space)
- Special tokens: , , , ,
- Trained from scratch on the training corpus
Usage
Training Philosophy
This model was trained entirely from scratch — no fine-tuning, no transfer learning, no derivative weights. Every parameter is original.
License
MIT
Real-World Applications
🔧 Lightweight Code Autocomplete (Embedded / Browser)
At 22M parameters, this model runs on Raspberry Pi, mobile browsers via WebAssembly, or any edge device:
from transformers import pipeline
pipe = pipeline("text-generation", model="pinkelephantlimited/pink-elephant-22m")
pipe("def fibonacci(n):", max_new_tokens=40)
pipe("import pandas as pd
df = pd.read_csv('data.csv')
df.", max_new_tokens=40)
📝 Inline Text Completion for Low-Power Devices
pipe("The quick brown fox jumps", max_new_tokens=20)
🤖 CI/CD Pipeline for Automated PR Reviews
pipe("class UserModel:", max_new_tokens=60)
Why 22M? Loads in milliseconds, runs on any hardware, uses <100MB RAM. Ideal for real-time applications where latency matters more than fluency.