SupraLabs

Supra2-Medium-Base

README

License: apache-2.0

Let the model speak - a sample

Prompt: "Artificial intelligence (AI) is " Completion:

plaintext
Artificial intelligence (AI) is espoused by the AI community.
The AI community is a group of people who are interested in AI and are interested in the use of AI in the field of AI.
The goal of AI is to improve the quality of life of people in the field.
The aim of AI is the development of AI and the application of AI in a society.
The purpose of AI is that it can be used to improve the performance of the society.
It is a technology that is used to improve human intelligence.
The technology is used to make the human intelligence.

Evaluation & Benchmarks

All benchmarks were evaluated using the EleutherAI LM-Eval Harness.

Table with columns: Model, PIQA (acc_norm), HellaSwag (acc_norm), ARC-Easy (acc_norm), ARC-Challenge (acc_norm)
Model	PIQA (acc_norm)	HellaSwag (acc_norm)	ARC-Easy (acc_norm)	ARC-Challenge (acc_norm)
Supra-50M-Base (50M)	0.62	0.32	0.46	0.25
Supra2-Medium-Base (25M)	59.14	29.29	41.84	23.72
Supra2-100M-Base (100M)	0.65	0.36	0.48	0.25

Final Train Loss: 3.2469 (no eval loss available)

Note: The model shows strong performance relative to its size, particularly given the high token-per-parameter ratio. It even strongly competes with our previous 50M model!

Model Details

Table

Developed by	SupraLabs
Model type	Causal decoder-only transformer (Qwen3)
Language	English
Parameters	25.37M total / ~20M non-embedding
Training tokens	20B (~800 tokens per parameter)
Context length	1,024
Precision	bfloat16
License	Apache 2.0

Architecture

Table with columns: Hyperparameter, Value
Hyperparameter	Value
Hidden size	512
Layers	7
Attention heads	8 (MHA)
Head dim	64
Intermediate size (SwiGLU)	896
Vocab size	16,384
Positional encoding	RoPE θ=10,000
Normalization	RMSNorm, ε=10⁻⁶
Tied embeddings	Yes

Training Data

Table with columns: Source, Share, Approx. tokens
Source	Share	Approx. tokens
`HuggingFaceFW/fineweb-edu` (`sample-100BT`)	100%	20B

Documents were tokenized with the custom tokenizer_16k, concatenated into a flat uint16 token stream, and packed into contiguous 1,024-token chunks (no padding, no document masking — sequences may cross document boundaries).

Training Procedure

Table with columns: Setting, Value
Setting	Value
Optimizer	AdamW (fused), β₁=0.9, β₂=0.95, ε=10⁻⁸
Peak learning rate	3×10⁻³
LR schedule	Cosine with min LR (10% of peak)
Warmup steps	1,000
Micro batch size	32
Gradient accumulation	4
Effective batch	256 sequences = 262,144 tokens/step
Weight decay	0.1
Gradient clipping

Key Design Decisions

High token-per-parameter ratio: At 800 tokens/param, this model pushes the boundaries of data efficiency for small models
Compact vocabulary: 16K vocab reduces embedding overhead while maintaining coverage
Tied embeddings: Reduces parameter count and improves training stability
Multi-head attention (MHA): Unlike larger Supra2 models using GQA, this architecture uses standard MHA for simplicity at this scale
No sliding window: Full attention within the 1K context

Usage

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "SupraLabs/Supra2-Medium"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

prompt = "The future of artificial intelligence"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=512,
        do_sample=True,
        temperature=0.2,
        top_p=0.85,
        top_k=25,
        no_repeat_ngram_size=3,
        repetition_penalty=1.2,
    )

print(tokenizer.decode(out[0], skip_special_tokens=True))

Tokenizer notes

The model uses a custom 16,384-token vocabulary optimized for English web text. The tokenizer is shared across the Supra2 family's smaller models for consistency and efficient multi-task fine-tuning.

Intended Use

Intended:

Research on extreme parameter efficiency and data-efficient pretraining
Ultra-lightweight edge deployments where memory is severely constrained
Educational use for understanding transformer architectures at minimal scale
Starting point for domain-specific fine-tuning when compute resources are limited
Ablation studies on small model behavior and scaling laws

Not intended:

Production deployment for critical applications
Factual question answering or knowledge-intensive tasks
Long-form coherent generation beyond a few paragraphs
Non-English text (trained exclusively on English)
Any application requiring safety guarantees or alignment

Limitations and Bias

Very small. At 25M parameters, the model has extremely limited capacity. Expect frequent hallucinations, factual errors, repetitive outputs, and poor reasoning.
Base model. No RLHF, no safety tuning, no refusal behavior. It will continue whatever text you give it, including harmful or offensive prompts.
Web-derived data. FineWeb-Edu is a filtered CommonCrawl derivative and carries the biases, stereotypes, and factual errors of the open web.
Short context. Trained exclusively at 1,024 tokens. Extrapolation beyond this length is untested and likely degraded.
No document masking. Attention could cross document boundaries within a packed chunk, which slightly blurs document independence.
English only. Performance on other languages will be poor to non-existent.
High token-per-parameter ratio. While efficient, training at 800 tok/param means the model may be under-trained compared to models trained at lower ratios (e.g., 300 tok/param).

Performance Characteristics

Despite its tiny size, Supra2-Medium achieves surprisingly coherent short-form generation. Key observations:

Strengths: Basic grammar, simple factoids, short completions, pattern matching
Weaknesses: Complex reasoning, arithmetic, long-range coherence, factual accuracy, nuanced understanding
Best use case: Generating 1-3 sentence completions, simple text transformations, educational demonstrations

Future work includes instruction-tuned variants and exploration of even more efficient architectures at this scale.

© SupraLabs 2026

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

SupraLabs

Model Tree

Base

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Let the model speak - a sample

Prompt: "Artificial intelligence (AI) is " Completion:

plaintext
Artificial intelligence (AI) is espoused by the AI community.
The AI community is a group of people who are interested in AI and are interested in the use of AI in the field of AI.
The goal of AI is to improve the quality of life of people in the field.
The aim of AI is the development of AI and the application of AI in a society.
The purpose of AI is that it can be used to improve the performance of the society.
It is a technology that is used to improve human intelligence.
The technology is used to make the human intelligence.

Evaluation & Benchmarks

All benchmarks were evaluated using the EleutherAI LM-Eval Harness.

Table with columns: Model, PIQA (acc_norm), HellaSwag (acc_norm), ARC-Easy (acc_norm), ARC-Challenge (acc_norm)
Model	PIQA (acc_norm)	HellaSwag (acc_norm)	ARC-Easy (acc_norm)	ARC-Challenge (acc_norm)
Supra-50M-Base (50M)	0.62	0.32	0.46	0.25
Supra2-Medium-Base (25M)	59.14	29.29	41.84	23.72
Supra2-100M-Base (100M)	0.65	0.36	0.48	0.25

Final Train Loss: 3.2469 (no eval loss available)

Note: The model shows strong performance relative to its size, particularly given the high token-per-parameter ratio. It even strongly competes with our previous 50M model!

Model Details

Table

Developed by	SupraLabs
Model type	Causal decoder-only transformer (Qwen3)
Language	English
Parameters	25.37M total / ~20M non-embedding
Training tokens	20B (~800 tokens per parameter)
Context length	1,024
Precision	bfloat16
License	Apache 2.0

Architecture

Table with columns: Hyperparameter, Value
Hyperparameter	Value
Hidden size	512
Layers	7
Attention heads	8 (MHA)
Head dim	64
Intermediate size (SwiGLU)	896
Vocab size	16,384
Positional encoding	RoPE θ=10,000
Normalization	RMSNorm, ε=10⁻⁶
Tied embeddings	Yes

Training Data

Table with columns: Source, Share, Approx. tokens
Source	Share	Approx. tokens
`HuggingFaceFW/fineweb-edu` (`sample-100BT`)	100%	20B

Training Procedure

Table with columns: Setting, Value
Setting	Value
Optimizer	AdamW (fused), β₁=0.9, β₂=0.95, ε=10⁻⁸
Peak learning rate	3×10⁻³
LR schedule	Cosine with min LR (10% of peak)
Warmup steps	1,000
Micro batch size	32
Gradient accumulation	4
Effective batch	256 sequences = 262,144 tokens/step
Weight decay	0.1
Gradient clipping

Key Design Decisions

High token-per-parameter ratio: At 800 tokens/param, this model pushes the boundaries of data efficiency for small models
Compact vocabulary: 16K vocab reduces embedding overhead while maintaining coverage
Tied embeddings: Reduces parameter count and improves training stability
Multi-head attention (MHA): Unlike larger Supra2 models using GQA, this architecture uses standard MHA for simplicity at this scale
No sliding window: Full attention within the 1K context

Usage

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "SupraLabs/Supra2-Medium"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

prompt = "The future of artificial intelligence"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=512,
        do_sample=True,
        temperature=0.2,
        top_p=0.85,
        top_k=25,
        no_repeat_ngram_size=3,
        repetition_penalty=1.2,
    )

print(tokenizer.decode(out[0], skip_special_tokens=True))

Tokenizer notes

The model uses a custom 16,384-token vocabulary optimized for English web text. The tokenizer is shared across the Supra2 family's smaller models for consistency and efficient multi-task fine-tuning.

Intended Use

Intended:

Research on extreme parameter efficiency and data-efficient pretraining
Ultra-lightweight edge deployments where memory is severely constrained
Educational use for understanding transformer architectures at minimal scale
Starting point for domain-specific fine-tuning when compute resources are limited
Ablation studies on small model behavior and scaling laws

Not intended:

Production deployment for critical applications
Factual question answering or knowledge-intensive tasks
Long-form coherent generation beyond a few paragraphs
Non-English text (trained exclusively on English)
Any application requiring safety guarantees or alignment

Limitations and Bias

Very small. At 25M parameters, the model has extremely limited capacity. Expect frequent hallucinations, factual errors, repetitive outputs, and poor reasoning.
Base model. No RLHF, no safety tuning, no refusal behavior. It will continue whatever text you give it, including harmful or offensive prompts.
Web-derived data. FineWeb-Edu is a filtered CommonCrawl derivative and carries the biases, stereotypes, and factual errors of the open web.
Short context. Trained exclusively at 1,024 tokens. Extrapolation beyond this length is untested and likely degraded.
No document masking. Attention could cross document boundaries within a packed chunk, which slightly blurs document independence.
English only. Performance on other languages will be poor to non-existent.
High token-per-parameter ratio. While efficient, training at 800 tok/param means the model may be under-trained compared to models trained at lower ratios (e.g., 300 tok/param).

Performance Characteristics

Despite its tiny size, Supra2-Medium achieves surprisingly coherent short-form generation. Key observations:

Strengths: Basic grammar, simple factoids, short completions, pattern matching
Weaknesses: Complex reasoning, arithmetic, long-range coherence, factual accuracy, nuanced understanding
Best use case: Generating 1-3 sentence completions, simple text transformations, educational demonstrations

Future work includes instruction-tuned variants and exploration of even more efficient architectures at this scale.