CodeSoft

sorbet-v2-25m

README

License: apache-2.0

Architecture graph

~25M-parameter Qwen2-style decoder LM, warm-started from Sorbet-25M and continued-trained in two legs totaling ~12hrs on a single RTX 5060 Ti (16GB). Scores at or above Sorbet-25M on every benchmark below while cutting validation perplexity by ~6%.

Architecture

Identical to Sorbet-25M: stock Qwen2 throughout, no custom code paths, natively supported by both transformers and llama.cpp.

Table

Params	25,185,920 (~87% non-embedding)
Layers / hidden	14 / 384
Attention	GQA 6 heads / 2 KV heads, RoPE θ=100k
FFN	1024 (SwiGLU)
Context	4096
Vocab	8,192 custom byte-level BPE (tied embeddings)
Precision	bf16

Training

v2 continues the v1 checkpoint through two training runs:

Table with columns: Leg, Data mix (tokens), LR schedule
Leg	Data mix (tokens)	LR schedule
cpt2	fineweb-edu 70% / infiwebmath 10% / DCLM-baseline 20%, 0.8B tok	cosine, 8-bit AdamW
v2-final	FineWeb-HQ 65% / DCLM-baseline 20% / FineMath-4+ 15%, 1.7B tok	cosine peak 1e-4, fp32 AdamW

Block-shuffled at 131,072 tok/step.

Benchmarks

All numbers zero-shot via lm-evaluation-harness, bf16, identical settings across checkpoints.

Table with columns: Task, n, Random, acc, acc_norm
Task	n	Random	acc	acc_norm
HellaSwag	10,042	25%	26.55 ±0.44	26.63 ±0.44
ARC-easy	2,376	~25%	30.30 ±0.94	29.92 ±0.94
ARC-challenge	1,172	~25%	18.60 ±1.14	22.44 ±1.22
PIQA

Notes:

Every score is at or above the Sorbet-25M baseline within error bars.
ArithMark-3.0 (AxiomicLabs/Arithmark-3.0) remains the strongest relative result (+8 pts over random), consistent with the math share of the pretraining mix.
ARC-challenge raw accuracy sits below chance due to a length bias in unnormalized scores; acc_norm is the meaningful metric there.

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "CodeSoft/sorbet-v2-25m"
model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16").to("cuda")
tok = AutoTokenizer.from_pretrained(repo, subfolder="tokenizer")
ids = tok("Once upon a time", return_tensors="pt").input_ids.cuda()
print(tok.decode(model.generate(ids, max_new_tokens=64)[0]))

Limitations

Expect shallow world knowledge and weak performance on knowledge-heavy benchmarks due to the model's small parameter count and limited training budget.

License

Apache-2.0.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

CodeSoft

Model Tree

Base

this model

Input Modalities

Text

Output Modalities