MaliosDark

Nexus-Erebus-135M

Deploy Dedicated

README

License: other

Model architecture

Decoder-only transformer with tied input/output embeddings, grouped-query attention (GQA), and rotary position embeddings (RoPE).
Hidden size 640, 25 layers, 10 attention heads (5 key/value heads), context length 1024.
Vocabulary of 32,000, byte-level BPE with digit isolation: every digit is its own token. This single choice is what makes small-scale arithmetic work, because the model reasons over individual digits instead of arbitrary multi-digit chunks.

Training data

Pretraining (from scratch, 2.66B tokens) on a 2025 English web mixture, by token share:

Ultra-FineWeb, a re-filtered FineWeb successor (0.45)
DCLM / DataComp-LM baseline web (0.30)
Nemotron-CC, public sample (0.17)
Synthetic integer arithmetic (0.05)
Templated reasoning and word problems (0.03)

Fine-tuning corpus, built only from official training splits (no validation or test item is ever seen, so evaluation is contamination-free by construction):

ARC-Easy, ARC-Challenge, OpenBookQA, SciQ, QASC, CommonsenseQA, HellaSwag, PIQA, WinoGrande.
150k synthetic arithmetic completions.

Training procedure

Pretraining used bf16 with a cosine schedule. Fine-tuning ran for 2 epochs at a low learning rate.

The important detail is the fine-tuning format. Every example is rendered in completion form, that is context + correct continuation (for example 8 + 5 = 13), rather than a letter-answer form such as Question: ... Answer: D. This is deliberate: the benchmarks are scored by the log-probability the model assigns to each candidate continuation, so the thing worth optimizing is how likely the model makes the correct continuation. A model trained to emit a letter learns to pick a letter, not to prefer the correct completion, and lands near chance on the real scoring. Training in completion form aligns the objective with the evaluation.

Evaluation

All benchmarks are run 0-shot and scored by log-likelihood multiple choice: each candidate answer is appended to the context and the highest-scoring continuation is selected.

Table with columns: Benchmark, Metric, Score
Benchmark	Metric	Score
HellaSwag	acc_norm	30.61
ARC-Easy	acc_norm	48.61
ARC-Challenge	acc_norm	25.43
PIQA	acc	64.53
ArithMark-2	acc (length-normalized LL)	66.68

Aggregate = ( HellaSwag + (ARC-Easy + ARC-Challenge) / 2 + PIQA + ArithMark-2 ) / 4 = 49.71

ArithMark-2 is the AxiomicLabs/ArithMark-2.0 held-out arithmetic set; it is a synthetic benchmark and is never used for training.

A note on metric choice (for reproducibility)

HellaSwag and ARC are conventionally reported with length-normalized accuracy (acc_norm), not raw accuracy (acc). Under raw accuracy, longer correct answers are penalized simply for having more tokens, which understates these tasks and can even push ARC-Challenge below random chance. We therefore report acc_norm for HellaSwag and ARC, and acc for PIQA, following common practice. Comparing an acc number for one model against an acc_norm number for another produces non-comparable results; all figures above use the conventions above.

The Isabel Method

The Isabel Method is a small-model recipe with five parts:

a digit-isolating tokenizer so arithmetic is learnable at small scale,
from-scratch pretraining on recent, high-quality web corpora,
arithmetic bootstrapping mixed into pretraining,
contamination-free fine-tuning on official training splits only, in completion format, and
evaluation that matches the scoring the benchmarks actually use.

How to use

python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("MaliosDark/Nexus-Erebus-135M")
model = AutoModelForCausalLM.from_pretrained("MaliosDark/Nexus-Erebus-135M")

prompt = "8 + 5 ="
ids = tok(prompt, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=8)
print(tok.decode(out[0], skip_special_tokens=True))

Intended use

Research on small language models, digit-aware tokenization for arithmetic, and from-scratch training recipes.
English text generation and multiple-choice reasoning at the 135M scale.

Limitations

English only, with a context length of 1024 tokens.
As a compact research model, outputs should be verified before use in factual or downstream applications.

Creator: Malios Dark (Ideoa Labs).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

MaliosDark

Model Tree

Base

this model

Input Modalities

Text

Output Modalities