VantoraLabs

Vantora-Micro

Model Details

Table with columns: Property, Value
Property	Value
Architecture	LlamaForCausalLM (pure transformer)
Parameters	9,800
Vocab size	1024 (ByteLevel BPE)
Hidden size (d_model)	8
Intermediate size	22 (MLP ratio 2.77)
Hidden layers	2
Attention heads	1
Head dim	8
Context length	512
RoPE theta	10000.0
RMSNorm eps	1e-6
Tied embeddings	Yes
Dtype	float32

Training

Data: first 100M tokens of HuggingFaceFW/fineweb-edu sample-10BT
Epochs: 1 (100M total tokens seen)
Batch: 128 × seq 256 (3,051 steps)
Optimizer: AdamW, lr 5e-3, cosine schedule + 15% warmup
Grad clip: 1.0, seed 42
Hardware: NVIDIA GTX 750 (Maxwell, 4 GB VRAM)
Time: ~4.3 minutes

Benchmark: BananaMind Base Bench 1.1

Evaluated with the official BananaMind benchmark.py runner (official_complete_run: true, exact 350-item split, SHA-256 verified).

Table with columns: Metric, Value
Metric	Value
Overall Elo	810
Accuracy	26.00% (91/350)
Weighted accuracy	25.48%

Table with columns: Category, Elo, Accuracy
Category	Elo	Accuracy
Language Completion	919	52.0%
Commonsense	658	16.0%
World Knowledge	702	20.0%
Context Tracking	665	12.0%
Quantitative	875	26.0%
Logical Reasoning	839

⚠️ Length-bias caveat on Code Completion

The Code Completion score (Elo 982, 34%) is not evidence the model can code. It is a benchmark artifact:

In the code_completion category, the correct answer is the longest continuation 68% of the time (vs 12-34% in every other category).
This model has a length bias: it picks the longest continuation more often than random, because its token distribution is near uniform and longer sequences accumulate more probability.
The two effects line up, so the length bias coincidentally matches the correct answer most of the time.

The BananaMind README itself warns: "Mean token log-probability reduces direct continuation-length bias but does not eliminate every tokenizer-dependent effect." Treat the Code Completion Elo as a length-bias artifact, not a real coding skill.

vs. Vantora-Micro-Hybrid (same size/data)

Table with columns: Vantora-Micro, Vantora Micro Hybrid
	Vantora-Micro	Vantora Micro Hybrid
Params	9,800	11,256
Overall Elo	810	863
Accuracy	26.00%	30.29%
Val loss (edu)	4.9097	4.8584
Training time	~4.3 min	~49.5 min

The hybrid edges out this model by +53 Elo and +4.3% accuracy, but most of that gap comes from the length-bias artifact on Code Completion, not real reasoning. On PIQA / HellaSwag / ARC-Easy the two are within noise (0.5-2%).

Why this model is the practical choice

For a 10K-param model on a 100M-token slice of web text, the pure transformer is the better tradeoff:

11.5× faster to train (4.3 min vs 49.5 min on the same GTX 750).
Within noise of the hybrid on every benchmark that measures real ability.
No custom architecture, no trust_remote_code, loads with stock AutoModelForCausalLM.

The hybrid's SSM sequence memory was a clear win on TinyStories (where narrative memory mattered), but on this benchmark the extra training time buys almost nothing. This model gets the same result in a fraction of the time.

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("VantoraLabs/Vantora-Micro")
tokenizer = AutoTokenizer.from_pretrained("VantoraLabs/Vantora-Micro")

prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Files

markdown
config.json              # LlamaForCausalLM config
model.safetensors        # 9,800-param weights
tokenizer.json          # ByteLevel BPE (1024 vocab)
tokenizer_config.json   # tokenizer settings
special_tokens_map.json # special token mapping
generation_config.json  # generation defaults

Notes

This is an extremely small model — it is a research artifact for studying scaling laws and architecture comparisons at the sub-10K parameter scale, not a production language model. Its BananaMind score (Elo 810) is near the four-choice random baseline (25%), as expected for a model this size.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

VantoraLabs

Model Tree

Base

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

Model Details

Table with columns: Property, Value
Property	Value
Architecture	LlamaForCausalLM (pure transformer)
Parameters	9,800
Vocab size	1024 (ByteLevel BPE)
Hidden size (d_model)	8
Intermediate size	22 (MLP ratio 2.77)
Hidden layers	2
Attention heads	1
Head dim	8
Context length	512
RoPE theta	10000.0
RMSNorm eps	1e-6
Tied embeddings	Yes
Dtype	float32

Training

Data: first 100M tokens of HuggingFaceFW/fineweb-edu sample-10BT
Epochs: 1 (100M total tokens seen)
Batch: 128 × seq 256 (3,051 steps)
Optimizer: AdamW, lr 5e-3, cosine schedule + 15% warmup
Grad clip: 1.0, seed 42
Hardware: NVIDIA GTX 750 (Maxwell, 4 GB VRAM)
Time: ~4.3 minutes

Benchmark: BananaMind Base Bench 1.1

Evaluated with the official BananaMind benchmark.py runner (official_complete_run: true, exact 350-item split, SHA-256 verified).

Table with columns: Metric, Value
Metric	Value
Overall Elo	810
Accuracy	26.00% (91/350)
Weighted accuracy	25.48%

Table with columns: Category, Elo, Accuracy
Category	Elo	Accuracy
Language Completion	919	52.0%
Commonsense	658	16.0%
World Knowledge	702	20.0%
Context Tracking	665	12.0%
Quantitative	875	26.0%
Logical Reasoning	839

⚠️ Length-bias caveat on Code Completion

The Code Completion score (Elo 982, 34%) is not evidence the model can code. It is a benchmark artifact:

In the code_completion category, the correct answer is the longest continuation 68% of the time (vs 12-34% in every other category).
This model has a length bias: it picks the longest continuation more often than random, because its token distribution is near uniform and longer sequences accumulate more probability.
The two effects line up, so the length bias coincidentally matches the correct answer most of the time.

vs. Vantora-Micro-Hybrid (same size/data)

Table with columns: Vantora-Micro, Vantora Micro Hybrid
	Vantora-Micro	Vantora Micro Hybrid
Params	9,800	11,256
Overall Elo	810	863
Accuracy	26.00%	30.29%
Val loss (edu)	4.9097	4.8584
Training time	~4.3 min	~49.5 min

Why this model is the practical choice

For a 10K-param model on a 100M-token slice of web text, the pure transformer is the better tradeoff:

11.5× faster to train (4.3 min vs 49.5 min on the same GTX 750).
Within noise of the hybrid on every benchmark that measures real ability.
No custom architecture, no trust_remote_code, loads with stock AutoModelForCausalLM.

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("VantoraLabs/Vantora-Micro")
tokenizer = AutoTokenizer.from_pretrained("VantoraLabs/Vantora-Micro")

prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Files

markdown
config.json              # LlamaForCausalLM config
model.safetensors        # 9,800-param weights
tokenizer.json          # ByteLevel BPE (1024 vocab)
tokenizer_config.json   # tokenizer settings
special_tokens_map.json # special token mapping
generation_config.json  # generation defaults