Evicka

Hanse2-100M-Base

README

License: apache-2.0

Highlights

99.1M parameters, trained from scratch
19.999B pretraining tokens (~201.7 tokens per parameter)
German + English, approximately 56% / 44% by planned token share
2K → 4K → 8K progressive context curriculum
32K custom byte-level BPE tokenizer
Single-GPU training on an AMD Radeon RX 9070 XT 16 GB
Reproducible project code for tokenizer training, pretraining, and inference
Apache-2.0 release

Model overview

Table

Model type	Decoder-only causal language model
Architecture	Llama-style Transformer
Parameters	99,144,320
Languages	German, English
Vocabulary	32,000
Training tokens	19,999,752,192
Tokens / parameter	~201.7
Configured context	8,192 tokens
Training precision	bfloat16
Position encoding	RoPE, θ = 10,000
Embeddings	Tied input/output embeddings
Instruction tuned	No
Safety aligned	No

Architecture

Table with columns: Hyperparameter, Value
Hyperparameter	Value
Hidden size	640
Layers	16
Attention heads	10
KV heads	2
Head dimension	64
Intermediate size	2,048
Vocabulary size	32,000
RoPE θ	10,000
Tied embeddings	Yes

Pretraining

Pretraining used three consecutive phases while keeping the effective optimizer batch at approximately 131,072 tokens per optimizer step.

Table with columns: Phase, Purpose, Tokens, Context, Micro batch, Grad. accum., Learning rate
Phase	Purpose	Tokens	Context	Micro batch	Grad. accum.	Learning rate
1	Broad bilingual pretraining	~15B	2,048	2	32	1% warmup → constant `6e-4`
2	Quality-focused annealing	~4B	4,096	1

Hanse2 pretraining curriculum

Data mixture

Approximate token share over the full curriculum:

Table with columns: Source, Share
Source	Share
German FineWeb	43.5%
English FineWeb-Edu	39.0%
German FineWiki	12.5%
English FineWiki	5.0%

This corresponds to approximately 56% German / 44% English.

Optimization

Table with columns: Setting, Value
Setting	Value
Optimizer	AdamW
β1 / β2	0.9 / 0.95
Weight decay	0.1
Gradient clipping	1.0
Effective batch	~131,072 tokens / optimizer step
Training precision	bfloat16
Framework	PyTorch + Transformers
Attention	SDPA / AOTriton on ROCm
`torch.compile`

Final held-out evaluation loss after Phase 3 was approximately 2.825 on the fixed bilingual evaluation split.

Tokenizer

Hanse2 uses a custom 32,000-token byte-level BPE tokenizer trained on a German–English mixture.

Several tokens were reserved for later instruction/tool post-training. They were not used as a chat format during base pretraining, and this checkpoint should not be prompted with an assumed chat template.

Evaluation

The following model-comparison results were produced with the EleutherAI LM Evaluation Harness using the same local setup for Hanse2-100M and Supra-50M:

0-shot
no chat template
bfloat16
identical task implementations
identical random seeds

Table with columns: Benchmark, Metric, Hanse2-100M, Supra-50M
Benchmark	Metric	Hanse2-100M	Supra-50M
ARC Easy	`acc_norm`	0.4306	0.4609
ARC Challenge	`acc_norm`	0.2415	0.2534
HellaSwag	`acc_norm`	0.3122	0.3171
WinoGrande

Hanse2 remains close to Supra-50M on several English benchmarks while substantially improving the German linguistic evaluation.

Long-context diagnostic

The model is configured for 8,192 tokens and produces finite forward passes at ~7.7K tokens. This does not imply reliable retrieval across the full context window.

A small base-LM continuation test placed a one-token identifier earlier in a synthetic context and measured its next-token rank against nine decoys.

Table with columns: Context, Needle position, Accuracy, Mean rank, Mean margin
Context	Needle position	Accuracy	Mean rank	Mean margin
512	10%	1.00	1.00	10.08
512	50%	1.00	1.00	10.35
512	90%	1.00	1.00	9.31
1,024

The model can process long inputs, but retrieval degrades when relevant information is several thousand tokens away from the prediction point. Treat this as a small synthetic diagnostic, not a standardized long-context benchmark.

Quickstart

Transformers

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Evicka/Hanse2-100M-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

prompt = "Artificial intelligence is"
inputs = tokenizer(
    prompt,
    return_tensors="pt",
    return_token_type_ids=False,
).to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=150,
        do_sample=True,
        temperature=0.5,
        top_k=25,
        top_p=0.9,
        repetition_penalty=1.2,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

[!NOTE] This is a completion model. Prompts such as sentence beginnings, paragraphs, or documents to continue are more appropriate than chat-style system/user/assistant messages.

Reproducibility

The repository includes the core scripts needed to reproduce or adapt the project:

Table with columns: File, Purpose
File	Purpose
`train_tokenizer.py`	Train the custom byte-level BPE tokenizer
`train.py`	Run the three-phase pretraining pipeline
`inference.py`	Minimal local text generation

The training pipeline includes exact token-budget accounting, resumable checkpoints, deterministic held-out evaluation, and separate context/batch settings for each phase.

The initial training/tokenizer code was based on the Apache-2.0-licensed Supra-50M training scripts and was substantially rewritten and extended for Hanse2. No Supra model weights were used to initialize Hanse2.

Re-running the scripts does not by itself guarantee bit-identical reproduction across different hardware, driver, PyTorch, ROCm, dataset revisions, or nondeterministic kernels.

Intended use

Hanse2-100M-Base is intended primarily for:

research on small language models
German–English pretraining experiments
bilingual data-mixture studies
tokenizer and architecture experiments
progressive context-extension experiments
benchmarking and ablations
continued pretraining
supervised fine-tuning / post-training research
education and consumer-hardware experimentation

Limitations

At ~99M parameters, Hanse2 is a small research model. Expect:

weak factual recall and hallucinations
weak arithmetic and multi-step reasoning
repetition or topic drift
brittle knowledge of specific entities
limited long-range coherence and retrieval
unstable behavior outside its training distribution

The checkpoint has no instruction tuning, preference tuning, RLHF, refusal training, or production safety guardrails. It may generate inaccurate, biased, offensive, harmful, or otherwise undesirable continuations.

The configured 8K context length should not be interpreted as reliable 8K retrieval capability; see the long-context diagnostic above.

License

Released under the Apache License 2.0. See LICENSE.

Acknowledgements

Hugging Face — Transformers, Datasets, Tokenizers, FineWeb, FineWeb-Edu, and FineWiki
EleutherAI — LM Evaluation Harness
SupraLabs — original Apache-2.0 Supra-50M training/tokenizer scripts used as the starting point for the project code

Citation

If you use Hanse2-100M-Base in a project, a link to the model repository is appreciated.

bibtex
@misc{brauer2026hanse2base100m,
  author = {Erik Brauer},
  title  = {Hanse2-100M-Base},
  year   = {2026},
  url    = {https://huggingface.co/Evicka/Hanse2-100M-Base}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.