SupraLabs

SupraNeo-4M

README

License: apache-2.0

Architecture

Table

Parameters	4,070,240 (3.41M non-embedding, 83.9%)
Architecture	Qwen3 (`Qwen3ForCausalLM`)
Hidden size	160
Layers	12
Attention heads	4 (head_dim 40)
KV heads	2 (GQA 2:1)
MLP intermediate	432 (SwiGLU)
Vocabulary	4,096 (custom BPE, ~3.2 chars/token)
Context length	1,024
Normalization	RMSNorm + QK-Norm
Embeddings	tied
Precision	float32

No custom modeling code — trust_remote_code is not required.

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("DedeProGames/SupraNeo-4M")
model = AutoModelForCausalLM.from_pretrained("DedeProGames/SupraNeo-4M")

ids = tok("The main reason that", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=40, temperature=0.8, top_p=0.9, do_sample=True)
print(tok.decode(out[0]))

This is a base model with no chat template. apply_chat_template will fail by design.

Training

Pre-trained from scratch on a single NVIDIA L4 over 5B tokens (~1,230 tokens per parameter).

Table

Data	84% HuggingFaceFW/fineweb-edu filtered to `int_score ≥ 4`, 16% mlfoundations/dclm-baseline-1.0
Schedule	WSD, peak LR 4e-3, 1,500-step warmup, 1-sqrt decay over the final 20%
Anneal	last 20% on the FineWeb-Edu subset only, context extended 512 → 1,024
Batch	65,536 tokens/step
Optimizer	AdamW (β 0.9/0.95, wd 0.1 on 2D params, grad clip 1.0), z-loss 1e-4
Init	residual branches scaled by 1/√(2L)

The heavy FineWeb-Edu weighting is deliberate. DCLM-baseline was tuned to win reasoning benchmarks in the 1–7B range, and none of that transfers at 4M — what a model this size can learn is register and local fluency, and FineWeb-Edu's uniform expository prose is far easier to model.

Evaluation — BananaMind Base Bench 1.1

350 items, 4-way continuation-likelihood, add_special_tokens=False, no BOS, selection by highest conditional mean log-prob, Elo by weighted MLE with a prior of 4 games at 1000. Official run: dataset checksum verified, schema verified, 0 truncated contexts, 0 truncated continuations.

Table with columns: Category, Accuracy, z vs chance, Elo, Sig.
Category	Accuracy	z vs chance	Elo	Sig.
Language Completion	56.0%	+5.06	963	✅
Logical Reasoning	38.0%	+2.12	978	✅
World Knowledge	34.0%	+1.47	820

Table

Overall Elo	868
Chance floor	805
Above floor	+63
Raw accuracy	33.4% (95% CI 28.5–38.4%)
z vs chance	+3.64 — significant

By difficulty: easy 39.3%, medium 29.1%, hard 31.9%.

Reading these numbers honestly

The aggregate is significantly above chance, but the signal is concentrated in one place. Language Completion at 56% (+5.06σ) is the only strongly separated category, and that is exactly what a 4M model should be able to do: local grammatical and register plausibility. Logical Reasoning clears the bar marginally. The remaining five categories sit within noise, and Quantitative and Code Completion land at or slightly below chance — this model has no arithmetic or code capability, and the card should not be read as claiming otherwise.

The medium/hard inversion (29.1% vs 31.9%) is noise at n≈117, not evidence that harder items are easier.

BananaMind Base Bench was calibrated for the 65M–100M+ range. At 4M, with 350 four-way items, the detection floor at 1.96σ is roughly 29.5% accuracy — most of this benchmark simply lacks resolution here. For tracking progress at this scale, bits-per-byte on held-out text and BLiMP are the metrics with actual sensitivity. HellaSwag, PIQA and ARC are not reported because they sit at chance and measure nothing.

Limitations

SupraNeo-4M produces grammatical, register-consistent English with coherence over one to two sentences and topical drift beyond that. It has no factual reliability, no arithmetic, no code ability, and no instruction following. The 4,096-token vocabulary means it compresses text ~35% less efficiently than a standard 32k tokenizer, and its outputs are not comparable to other models by raw cross-entropy — use bits-per-byte.

This is a research artifact for studying small-scale pretraining, vocabulary budgets, and data mixtures. It is not intended for deployment.

Carbon footprint

Training was estimated to emit 0.46 kg CO₂ eq. — a single L4 for the duration of the run. For reference, that is roughly the footprint of driving a passenger car about 2.5 kilometers.

by DedeProGames

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

SupraLabs

Model Tree

Base

this model

Input Modalities

Text

Output Modalities