Supra-1.5-Base-1b_CPT_Ext_EXP

Intended Repository

SupraLabs/Supra-1.5-50M-Base

Architecture

The model keeps the original Supra-50M architecture and tokenizer:

• Architecture: LlamaForCausalLM
• Parameters: about 50M
• Vocabulary: 32,000
• Hidden size: 512
• Layers: 12
• Attention heads: 8
• KV heads: 4
• Context target: 5,120 tokens
• Tokenizer: original Supra byte-level BPE tokenizer

Continued Pretraining Objective

This is CPT, not instruction fine-tuning. Training uses packed raw text with standard causal language-modeling loss:

• labels = input_ids
• all non-pad tokens are trained
• no response-only masking
• no system/user/assistant masking
• no LoRA adapters in the default run

Data Mix

The current local training mix prepared for this run is:

• 3,000,000,062 CPT tokens
• 600,004,516 JSON/tool/problem-solving tokens
• English-only-ish filtering with JSON, math, numbers, URLs, punctuation, and emojis allowed
• Long factual/general text packed near 5k tokens
• Short Q&A/problem-solving/ChatML-style text mixed into CPT as raw text

See BREAKDOWN.md and ATTRIBUTIONS.md for details.

Training

Launch locally from the workspace:

powershell
Copy code
.\launch_supra_1_5_cpt.ps1

The launcher opens a standalone PowerShell training terminal and writes logs to outputs/Supra-1.5-50M-Base/training_terminal.log.

Evaluation

After checkpoints are available:

powershell
Copy code
C:\Users\artig\.unsloth\studio\unsloth_studio\Scripts\python.exe .\eval_long_context.py