Training Data
Table with columns: Source, Tokens, Description| Source | Tokens | Description |
|---|
| FineWeb-Edu | 900M | High-quality educational web text |
| Ubuntu IRC | 600M | Technical support chat logs |
| StackExchange | 1.05B | Q&A from StackExchange network |
| DCLM | 300M | Filtered web text |
Total: ~2.85B tokens (6 epochs = ~17B tokens seen)
Model Architecture
Table with columns: Parameter, Value| Parameter | Value |
|---|
| Parameters | 125,847,552 |
| Layers | 12 |
| Hidden dim | 768 |
| FFN dim | 3072 (SwiGLU) |
| Attention heads | 12 |
| KV heads | 12 (MHA) |
| Vocab size | 16,384 |
| Context length | 1,024 |
| Position encoding | RoPE |
| Norm | RMSNorm |
| Tie embeddings |
Training Details
- Hardware: 8x H100 (Modal cloud)
- Framework: PyTorch + DDP
- Optimizer: AdamW (lr=6e-4, warmup 200M tokens, cosine decay)
- Mixed precision: bfloat16
- Total cost: ~$31
- Val perplexity: 15.06
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("applegrew/support-125M-slm-base")
tokenizer = AutoTokenizer.from_pretrained("applegrew/support-125M-slm-base")
prompt = "The VPN connection keeps dropping"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Special Tokens
This model uses custom chat tokens: <|bos|>, <|eos|>, <|pad|>, <|unk|>, <|system|>, <|user|>, <|assistant|>
SFT Version
For instruction following, use the SFT version: applegrew/support-125M-slm-sft