nkthebass

tinybrainbot-320mV2-instruct

README

License: apache-2.0

Model details

Table

Parameters	325,899,264 (~326M)
Architecture	Decoder-only transformer, pre-norm, RMSNorm, SwiGLU MLP, RoPE
Hidden size	1024
Layers	26
Attention heads	16 (query) / 4 KV heads (grouped-query attention)
FFN size	2816
Context length	1024
Vocabulary	32,000
Tokenizer	`tbb-32k-v2` — 32k BPE (67% English / 20% code / 13% math), with reserved `<think>`/`</think>` special tokens
Precision	trained in fp16 with an fp32 master copy (autocast)

Usage

Prompt with the chat format:

markdown
<|user|>
{user message}
<|end|>
<|assistant|>
{assistant reply}
<|end|>

Example (greedy):

markdown
<|user|>
What is the capital of France?
<|end|>
<|assistant|>
Paris.
<|end|>

The model gives concise direct answers and shows worked steps for arithmetic.

Prompting tips (important for good answers)

At ~326M parameters this is a small model and it is sensitive to phrasing. Ask direct, specific questions rather than open-ended imperatives — the difference is large:

Table with columns: Prompt, Output
Prompt	Output
✅ `What are the eight planets in the solar system?`	Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, and Neptune.
✅ `How many planets are in the solar system?`	Eight.
✅ `What is the capital of Japan?`	Tokyo.
⚠️ `list all of the planets in the solar system`	terse or empty

On short / empty replies (e.g. in LM Studio, Jan, Ollama): very short or empty answers on certain imperative phrasings — and in long multi-turn chats — are a model-capacity / instruction-coverage characteristic of a 326M model, not a GGUF or chat-format bug. The F16 GGUF is a faithful conversion: it produces token-for-token identical output to the fp16 transformers model (verified). Use the model's built-in chat template as-is — spaces between the role tags are correct (the SentencePiece tokenizer normalizes newlines to spaces, so <|user|> … <|end|> and the newline form encode to the same token IDs the model trained on).

Training procedure

training loss curve

Pretraining (blue, ~10B tokens) then instruct SFT (orange) — SFT starts at step 51,000.

Pretraining (base)

Pretrained from scratch on 10.03B tokens (51,000 steps, WSD schedule, peak LR 6e-4, two-phase broad → quality-anneal, final loss 1.436) across 2× Tesla V100-PCIE-16GB with PyTorch DDP (gloo), fp16 autocast + fused AdamW. Full pretraining details and the 13-source data mix are documented on the base model card.

Supervised fine-tuning (this model)

Table with columns: Hyperparameter, Value
Hyperparameter	Value
Steps	~1,433 (resumed from base @ 51,000)
Tokens	~94M
LR	1.5e-5, constant
Warmup	100 steps
Global batch	4 micro × 8 grad-accum × 2 GPUs × 1024 seq
Loss masking	assistant-only
Best checkpoint	step 52,000 (selected on eval, not last)

SFT data mix (sampling weights; sum = 9.5; math ≈ 15.8% of the mixture):

Table with columns: Source, Weight
Source	Weight
longdef-sft (~16k long / multi-step answers)	2.0
smoltalk	2.0
math-v2 (~70k verified worked examples)	1.5
qa-distill	1.0
lamini-instructions	1.0
soda-dialogues	1.0
greetings	0.5
reasoning-distill	0.5

math-v2 is ~70k programmatically-generated, tolerance-verified arithmetic worked examples (multiplication / division / decimals, with the final answer checked). It appears in both pretraining and SFT — see the note below.

On where the arithmetic comes from: the base and instruct models score almost identically on the GPT-3 Arithmetic suite (aggregate 31.4% vs 31.2%). That base/instruct parity suggests most arithmetic computation was acquired during pretraining, while SFT mainly shaped instruction-following and response format (the one sub-task where SFT clearly helps is composite / order-of-operations, 7.0% vs 3.7%). Pretraining also delivered far more math tokens in absolute terms — on the order of ~215M vs ~15M — despite SFT's higher math proportion.

Evaluation

Measured on our own log-likelihood MC harness (lm-eval style, fixed seed). Headline metric = acc_norm for HellaSwag/ARC/OpenBookQA, acc for WinoGrande/MMLU. Reference values are published lm-eval approximations — treat gaps under ~±2 points as ties.

vs the previous 303M instruct (full test sets)

Table with columns: Benchmark, n, 320M V2 instruct, 303M instruct
Benchmark	n	320M V2 instruct	303M instruct
HellaSwag	10042	34.5	30.7
ARC-Easy (acc_norm)	2376	49.3	47.6
ARC-Easy (raw acc)	2376	57.0	51.0
ARC-Challenge	1172	27.6	27.6

→ 5 wins, 1 tie, 0 losses over the previous generation.

vs reference models (headline metric)

Table with columns: Benchmark, 320M V2 instruct, GPT-2-124M, Pythia-410M, SmolLM-360M
Benchmark	320M V2 instruct	GPT-2-124M	Pythia-410M	SmolLM-360M
HellaSwag	34.5	31	34	54
ARC-Easy	49.3 (57 raw)	44	52	70
ARC-Challenge	27.6	22	24	37

→ Sweeps GPT-2-124M; ~5 wins + 1 draw vs Pythia-410M. SmolLM-360M (trained on ~600B aggressively-filtered tokens) remains the frontier for this size.

Training efficiency. These results come from ~10B pretraining tokens — roughly an order of magnitude fewer than the Pythia suite's ~300B. The Pythia-410M parity is therefore best read as a token-efficiency result (curated data + quality anneal) rather than a scale win.

Math — computation vs reasoning

The model was trained on arithmetic computation, not word-problem reasoning — the two benchmarks below show that split clearly.

GSM8K (grade-school word problems, full 1319-problem test, zero-shot chain-of-thought):

Table with columns: Model, GSM8K
Model	GSM8K
GPT-2-124M	~0%
320M V2 instruct	0.53%
Pythia-410M	~1–2%
SmolLM2-360M-Instruct	~3–5%

→ At the floor for a general-purpose model of this size and training mix; stronger sub-1B math-specialized models can score substantially higher. GSM8K rewards multi-step semantic reasoning, which this recipe did not target.

GPT-3 Arithmetic (Brown et al. 2020 protocol, exact-match, n=300/sub-task):

Table with columns: Sub-task, Accuracy
Sub-task	Accuracy
2-digit addition	99.0%
3-digit addition	94.0%
2-digit subtraction	49.3%
3-digit subtraction	42.7%
4-digit addition / subtraction	0.3% / 0.3%
5-digit addition / subtraction	0.0% / 0.0%
2-digit multiplication	21.7%
single-digit composite (order of ops)	7.0%

→ Strong through 3 digits, then a hard wall. 2–3-digit addition is near-solved (94–99%, with correct carrying), but 4+-digit accuracy collapses to ~0%: the model executes a fixed ~3-column addition routine and silently drops the higher place values — a length-generalization limit tied to the training distribution (math-v2 operands are ≤3 digits), not truncation (generations complete normally and end with a stated answer). Subtraction sits ~42–50% — it handles a−b when a>b but drops the sign on negative results. 2-digit multiplication ~22%; single-digit composite (order of operations) ~7%.

How to read this: the GPT-3 Arithmetic suite mainly probes exact symbolic computation and short-range algorithmic generalization; it should not be interpreted as evidence of strong mathematical reasoning (see GSM8K above). The two results together are the point: strong at computing, weak at reasoning.

Intended use & limitations

Intended use: a capable general chat assistant at ~326M scale, on-device / low-resource deployment, research on small-model SFT, and arithmetic computation.

Limitations:

Math reasoning (word problems, GSM8K/MATH) is at the floor — the model computes but does not reason through multi-step problems.
Negative-result subtraction is unreliable (drops the sign).
WinoGrande and MMLU sit near the random floor — consistent with the capacity and data limits of a ~326M model under this training recipe.
Trained predominantly on English; 1024-token context; no RLHF/safety tuning — outputs may be incorrect or inappropriate and should not be relied upon unchecked.

Hardware & framework

2× NVIDIA Tesla V100-PCIE-16GB · Windows · PyTorch DDP (gloo) · fp16 autocast (fp32 master) · fused AdamW · custom TinyBrainBot trainer.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.