Scores
Table with columns: v5.1-Instruct (this), 100M-v5-Base (its base), 360M-A75M-v5-Instruct (3.8x bigger) | v5.1-Instruct (this) | 100M-v5-Base (its base) | 360M-A75M-v5-Instruct (3.8x bigger) |
|---|
| IFEval prompt-level strict | 19.8 | | 25.1 |
| IFEval instruction-level strict | 33.0 | | 37.6 |
| Open SLM Leaderboard Intelligence Index | 23.11 | 26.07 | 27.82 |
| (raw acc) | 21.01 | | 26.07 |
IFEval was run with lm-eval and the chat template, greedy, with the <think> block removed before scoring. The Index uses
the full HellaSwag / ARC / PIQA sets plus ArithMark-3, 0-shot.
Table with columns: Benchmark (0-shot, full set), acc_norm, (raw acc)| Benchmark (0-shot, full set) | acc_norm | (raw acc) |
|---|
| HellaSwag | 35.11 | 31.20 |
| ARC-Easy | 55.01 | 58.63 |
| ARC-Challenge | 30.12 | 27.22 |
| PIQA | 65.67 | 64.20 |
| ArithMark-3 | 43.60 | 43.60 |
Quick start
from transformers import AutoModelForCausalLM, AutoTokenizerrepo = "nkthebass/tinybrainbot-100m-v5.1-instruct"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo)msgs = [{"role": "user", "content": "Write a short poem about autumn."}]ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")out = model.generate(ids, max_new_tokens=200, temperature=0.4) # other defaults from generation_config.jsonprint(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
GGUF (llama.cpp, LM Studio, Jan, Ollama): F16 (201 MB), Q8_0 (107 MB) and Q4_K_M (66 MB). The chat template is
embedded, chats tokenize exactly like transformers (line breaks included), and greedy output matches token for token.
llama-cli -m tinybrainbot-100m-v5.1-instruct-Q8_0.gguf -cnv --temp 0.4 --top-p 0.9 --repeat-penalty 1.15 --logit-bias 241-inf --logit-bias 255-inf
Recommended sampling: temperature 0.4, top-p 0.9, repetition penalty 1.15. The two --logit-bias flags block the byte
tokens that start every emoji: the tokenizer has no emoji pieces, so the model would otherwise spell them out byte by byte.
<think> blocks are plain text, not special tokens. Strip them if you only want the answer:
import re; answer = re.sub(r"<think>.*?</think>", "", reply, flags=re.S).strip()
What it does well (and doesn't)
- Conversation: short, friendly answers; it remembers names and details from earlier turns.
- Layout: numbered lists and multi-line poems.
- Math: it thinks before answering on 9 of 10 short math questions, and gets about half of them right.
- Not yet: it doesn't always follow exact counts ("give me 3 tips" can give 5), some practical answers drift off
topic, and at 100M parameters it makes up facts.
Unedited sample conversations, including mistakes, are in conversation_samples.md.
Training
- Pretraining. TinyBrainBot-100M-v5-Base: 29.6B tokens.
- SFT (0.39B tokens, 3,000 steps), one mix for every skill, the recipe from 360M-A75M-v5-Instruct, now with line breaks kept:
Table with columns: Part, Share, What| Part | Share | What |
|---|
| Chat | 40% | smol-smoltalk, multi-turn conversations, recall and follow-up dialogues, WildChat (refusals removed) |
| Math | 25% | column-routine arithmetic (wide numbers, negatives, 3x3 multiplication), cue-first word problems, generated scratchpad word problems |
| Reasoning | 25% | <think> reasoning traces, open-ended reasoning, safety-aware reasoning, with plain-chat ballast so it doesn't think on "Hi" |
| Instruction following | 10% | Tulu 3 persona instruction-following |
- DPO. 10,383 preference pairs (UltraFeedback, instruction-constraint pairs checked by code, and judge-ranked pairs),
beta 0.5, LR 1e-6, 2 epochs.
No benchmark training or test data was used, and no GSM8K-derived sets.
Limitations
- English only. It confabulates facts, so check anything important.
- Extra spaces are still collapsed, so it can't keep code indentation.
- 1,024-token context.