Intended Repository
SupraLabs/Supra-1.5-50M-Base
Architecture
The model keeps the original Supra-50M architecture and tokenizer:
- Architecture:
LlamaForCausalLM
- Parameters: about 50M
- Vocabulary: 32,000
- Hidden size: 512
- Layers: 12
- Attention heads: 8
- KV heads: 4
- Context target: 5,120 tokens
- Tokenizer: original Supra byte-level BPE tokenizer
Continued Pretraining Objective
This is CPT, not instruction fine-tuning. Training uses packed raw text with
standard causal language-modeling loss:
labels = input_ids
- all non-pad tokens are trained
- no response-only masking
- no system/user/assistant masking
- no LoRA adapters in the default run
Data Mix
The current local training mix prepared for this run is:
- 3,000,000,062 CPT tokens
- 600,004,516 JSON/tool/problem-solving tokens
- English-only-ish filtering with JSON, math, numbers, URLs, punctuation, and
emojis allowed
- Long factual/general text packed near 5k tokens
- Short Q&A/problem-solving/ChatML-style text mixed into CPT as raw text
See BREAKDOWN.md and ATTRIBUTIONS.md for details.
Training
Launch locally from the workspace:
.\launch_supra_1_5_cpt.ps1
The launcher opens a standalone PowerShell training terminal and writes logs to
outputs/Supra-1.5-50M-Base/training_terminal.log.
Evaluation
After checkpoints are available:
C:\Users\artig\.unsloth\studio\unsloth_studio\Scripts\python.exe .\eval_long_context.py