Benchmarks (EleutherAI lm-eval, 0-shot, acc_norm; WinoGrande/MMLU = acc)
Table with columns: Benchmark, This model, Supra2-100M-Base, Supra2-100M-Instruct| Benchmark | This model | Supra2-100M-Base | Supra2-100M-Instruct |
|---|
| ARC-Easy | 54.7 | 47.8 | 44.4 |
| ARC-Challenge | 30.1 | 24.8 | 24.7 |
| OpenBookQA | 34.0 | 32.0 | 30.4 |
| WinoGrande | 53.0 | 50.7 | 50.5 |
| PIQA | 66.2 | 65.5 | 64.4 |
| MMLU | 25.0 | 23.3 | 25.8 |
| HellaSwag | 32.6 | 36.0 | 35.9 |
6/7 vs Supra2-Base and 5/7 vs Supra2-Instruct. HellaSwag is the one benchmark where Supra leads.
Reproduce these numbers
EleutherAI lm-eval-harness v0.4.12, 0-shot, evaluated on the HF repo (do not eval the GGUF — llama.cpp's --multiple-choice path under-reports these tasks; the .hellaswag path is fine):
lm_eval --model hf \
--model_args pretrained=nkthebass/tinybrainbot-100m-v3-base,dtype=float32 \
--tasks hellaswag,arc_easy,arc_challenge,openbookqa,winogrande,piqa,mmlu \
--num_fewshot 0 --batch_size 32
Metrics: acc_norm for HellaSwag / ARC-Easy / ARC-Challenge / OpenBookQA / PIQA; acc for WinoGrande & MMLU.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("nkthebass/tinybrainbot-100m-v3-base")
model = AutoModelForCausalLM.from_pretrained("nkthebass/tinybrainbot-100m-v3-base")
ids = tok("The capital of France is", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=20)[0]))
GGUF
An F16 GGUF is included (tinybrainbot-100m-v3-base-f16.gguf) for llama.cpp / Ollama / LM Studio, with the correct add_space_prefix=false + leading-space template baked in for faithful tokenization.
Limitations
A 100M base model: strong on multiple-choice reasoning for its size, but open-ended generation is limited and can be factually unreliable. For chat use the instruct variant; for arithmetic use the math variant. Not aligned or safety-tuned.
Companion models: instruct · math.