The model uses the ChatML-style turn format of Qwen3. The tokenizer is the OpenEuroLLM 256K
tokenizer from its qwen3-tokens branch. That branch gives Qwen3 names to eight token IDs that the
base tokenizer had reserved:
Table with columns: Token, ID, Name in the base tokenizer| Token | ID | Name in the base tokenizer |
|---|
<|im_start|> | 3 | <start_of_turn> |
<|im_end|> | 4 | <end_of_turn> |
<tools> / </tools> | 13 / 14 | <unused_0> / <unused_1> |
<tool_response> / </tool_response> | 15 / 16 | <unused_2> / <unused_3> |
</think> / <think> | 17 / 18 | <unused_4> / <unused_5> |
Unchanged special tokens: <bos> = 1, <eos> = 2, <tool_call> / </tool_call> = 11 / 12,
<pad> = 262,144. All other token IDs are identical to the base tokenizer. On a sample of 3.9M
tokens of Dolci text, both tokenizers produced identical token IDs.
A rendered conversation looks like this:
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What is the capital of Finland?<|im_end|>
<|im_start|>assistant
<think>
</think>
The capital of Finland is Helsinki.<|im_end|>
- Each assistant turn ends with
<|im_end|>. This is the token that SFT taught the model to
emit. The pretraining terminator <eos> does not occur in the SFT data. Use <|im_end|> as the
stop token.
- The model emits an empty
<think> block. The template put a
<think>\n…\n</think>\n\n block at the start of the final assistant turn of every
conversation. Dolci Instruct has almost no reasoning traces (385 of 2,152,112 conversations have
</think> in an assistant message), so the block was nearly always empty. This is not a
reasoning model.
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "Neonkraft/oellm-9b-256k-theta64m-prelude-anneal300b-instruct-sft"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, dtype=torch.bfloat16, device_map="auto")
messages = [{"role": "user", "content": "Explain in two sentences why the sky is blue."}]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
enable_thinking=False,
return_dict=True,
return_tensors="pt",
).to(model.device)
output = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
generation_config.json lists <|im_end|> (4) and <eos> (2) as stop tokens, so generate()
stops at the end of the assistant turn.
Training data
The model was trained on all 2,152,112 conversations of the train split of
allenai/Dolci-Instruct-SFT
(revision main). Most of the data is English. The dataset card lists the sources and their licenses. The dataset is
licensed under ODC-BY and is intended for research and educational use under Ai2's
Responsible Use Guidelines.
Before training, the pipeline removed each conversation that had no supervised assistant token in
its first 32,768 tokens. It kept conversations where the 32,768-token limit cut a supervised span.
Training procedure
Full-parameter SFT with TRL SFTTrainer.
Loss masking. The loss covers only the assistant tokens that come after the last user message in
each conversation, including their closing <|im_end|>. System, user, and tool tokens, and all
assistant turns before the last user message, are context only. In a multi-turn conversation, only
the final assistant reply (or the chain of assistant turns after the last user message) is trained.
Packing. Conversations were packed with TRL's best-fit-decreasing (BFD) strategy into sequences
of up to 32,768 tokens, one packed sequence per device per step. If a conversation was longer than
32,768 tokens, only its first 32,768 tokens were used.
Table with columns: Hyperparameter, Value| Hyperparameter | Value |
|---|
| Epochs | 2 |
| Maximum sequence length | 32,768 |
| Global batch | 32 packed sequences (1 per device × 32 devices × 1 accumulation step), up to 1,048,576 tokens per step |
| Optimizer | AdamW (DeepSpeed), β₁ = 0.9, β₂ = 0.95, ε = 1e-8 |
| Weight decay | 0.0 |
| Peak learning rate | 8e-5 |
| Schedule | Linear warmup for 3% of steps, then linear decay to 0 |
| Gradient clipping | 1.0 |
| Precision |
Compute
Table with columns: Item, Value| Item | Value |
|---|
| Cluster | LUMI (CSC, Finland) |
| Accelerators | 4 nodes × 8 AMD MI250X GCDs = 32 GCDs |
| Software | ROCm 7.2.4, Python 3.12, PyTorch 2.9.1, TRL 1.7.0 (Singularity container) |
| Wall-clock time | 31 hours (about 990 GCD-hours) |
Reproducibility
Table with columns: Item, Value| Item | Value |
|---|
| Run name | oellm-9b-256k-theta64m-prelude-anneal300b-sft |
| Base model | birgermoell/oellm-9b-256k-theta64m-prelude-anneal300b @ b61957c3e22e6145ee7973f415c45047a0f9a108 |
| Tokenizer | openeurollm/tokenizer-256k @ 94bae05cf351c4feb435d6a73c2aa8edfd88ae04 (branch qwen3-tokens) |
| Dataset | allenai/Dolci-Instruct-SFT, split train @ |
Citation
The SFT data comes from the Olmo 3 release:
@misc{olmo2025olmo3,
title={Olmo 3},
author={Team Olmo},
year={2025},
eprint={2512.13961},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2512.13961},
}
Acknowledgements
Training ran on the LUMI supercomputer, hosted by CSC in Finland.