The architecture is Qwen3ForCausalLM with 9,101,947,904 parameters and the
OpenEuroLLM/Qwen3-style ChatML token mapping inherited from the parent.
Available tools are described in <tools>...</tools> in the system turn; the
assistant emits <tool_call>{"name": ..., "arguments": {...}}</tool_call>.
Tool replies are represented as <tool_response>...</tool_response> in a user
turn. An empty <think>\n\n</think>\n\n block precedes assistant responses
after the latest user query. <|im_end|> (token ID 4) terminates assistant
turns. The packaged chat_template.jinja implements this format.
Context and RoPE caveat: The parent advertises a 256K context and RoPE
theta 64,000,000. This continuation was trained with Transformers 4.57.3,
which resolved the parent configuration to an effective RoPE theta of 10,000
because the 64M value was stored in rope_parameters while that version's
Qwen3 rotary implementation read rope_theta. The exported configuration
explicitly sets both fields to 10,000 to preserve the behavior used during
training. Training sequences were at most 4,096 tokens. Long-context
performance, including 256K tool use, has not been validated. The
max_position_embeddings architecture field remains 262,144; it is not a
validated context-length claim for this checkpoint.
Training used BF16 computation. The saved full-parameter weights are FP32
(about 36 GB), so plan memory accordingly or explicitly load in BF16 for
inference.
Training data
Table with columns: Component, Training rows, Approximate rendered tokens, Source and license| Component | Training rows | Approximate rendered tokens | Source and license |
|---|
| Tool use | 36,642 | 20,031,319 | birgermoell/oellm-eu-tooluse-v1, sft/train, Apache-2.0 |
| General instruction replay | 31,695 | 20,032,863 | Dolci Instruct slice from the existing Anneal-300B data artifact; allenai/Dolci-Instruct-SFT, ODC-By-1.0 |
The tool-use dataset has 55,077 source rows and is derived from
glaiveai/glaive-function-calling-v2
(50,000 rows), Team-ACE/ToolACE
(3,184), and
NousResearch/hermes-function-calling-v1
(1,893). Its dataset card describes all three as Apache-2.0 and documents
their normalization into <tool_call> JSON. No examples from its grpo
configuration were used.
Preparation trimmed trailing non-assistant turns, filtered 14 overlength tool
rows, and removed 16,473 duplicate tool prompts keyed on both the system tool
schema and the first user request. A deterministic 5% prompt split held out
1,948 tool conversations (about 1,048,852 rendered tokens). Dolci replay
was selected from the parent training artifact until it matched the retained
tool training tokens. There were only 10 no-call conversations in the entire
tool-use source after normalization. This is a major limitation for learning
when to abstain from calling a tool.
The counts and checksums above come from the saved
training_data_manifest.json. They were
computed during data construction with the earlier chat template; the final
tool-use template adds an empty think block to intermediate tool calls, so
these token counts are approximate for the actual training serialization.
Training procedure
Table with columns: Item, Value| Item | Value |
|---|
| Method | Full-parameter SFT, assistant-only loss; system, user, and tool text are context |
| Packing | TRL best-fit decreasing, up to 4,096 tokens per packed sequence |
| Steps | 800 optimizer updates; approximately 104.09M packed tokens, 2.59 data epochs |
| Global batch | 32 packed sequences (32 GCDs × 1 sequence × 1 accumulation step) |
| Learning rate | Peak 2e-6; cosine schedule; 24 warmup steps |
| Optimizer | AdamW, β₁=0.9, β₂=0.95, ε=1e-8; no weight decay; gradient clip 1.0 |
| Runtime | 1 hour 28 minutes 51 seconds on 4 LUMI-G nodes, 8 AMD MI250X GCDs per node |
| Software |
The parent revision was
85bf18fb4f0bee6ac6270f06b1d1c6b3be200f31. The training data manifest
SHA-256 was caf837e99a93e2bcc9b71509980e60eba8895184216c541bc52548f5a92a3ac0.
The train Parquet SHA-256 was
01cbe9ecd71b7ef6f3b6c177ad06cc9692ff8489811eb13dc83b07a673d1f87f.
The repository state was based on
BirgerMoell/oellm-reasoning-training
commit 7b9f05a66bdfd6ca5b0d5e9c88cdabcc69098ca2 plus local uncommitted
tool-use changes. LUMI Slurm job: 22546173.
Usage
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer repo_id = "birgermoell/oellm-9b-anneal300b-tooluse-sft"tokenizer = AutoTokenizer.from_pretrained(repo_id)model = AutoModelForCausalLM.from_pretrained( repo_id, torch_dtype=torch.bfloat16, device_map="auto").eval() tools = [{"type": "function", "function": { "name": "get_weather", "description": "Get weather for a city", "parameters": {"type": "object", "properties": { "city": {"type": "string"}}, "required": ["city"]}}}]messages = [{"role": "user", "content": "What is the weather in Helsinki?"}]inputs = tokenizer.apply_chat_template( messages, tools=tools, add_generation_prompt=True, enable_thinking=False, return_dict=True, return_tensors="pt",).to(model.device)inputs.pop("token_type_ids", None) # needed with some Transformers versionswith torch.inference_mode(): output = model.generate(**inputs, max_new_tokens=192, do_sample=False)print(tokenizer.decode(output[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=False))
The dataset embedded tool signatures directly in system messages. Supplying
tools= as above renders the same tool wrapper through the packaged template.
Execute any returned tool call in your own application, then append the result
as a tool message and generate again. The model weights do not execute tools.
With Transformers 4.57.3, the tokenizer's saved metadata may trigger a
TypeError in AutoTokenizer. A compatible fallback is to load
tokenizer.json with PreTrainedTokenizerFast and set its chat_template
from chat_template.jinja (both files are in this repository). The training
and evaluation code use that fallback.
Evaluation and limitations
The paired held-out evaluation compares this checkpoint with its direct parent
using greedy decoding and at most 192 new tokens. It uses 100 callable prompts
from the held-out tool split and 100 untouched Dolci instruction prompts paired
with an irrelevant calculate_tip tool. The latter is a synthetic relevance
probe; the tool dataset itself contained too few no-call conversations for a
balanced held-out sample.
Table with columns: Metric (100 prompts per class), Parent SFT, Tool-use SFT| Metric (100 prompts per class) | Parent SFT | Tool-use SFT |
|---|
| Well-formed tool call on callable prompts | 0/100 (0%) | 99/100 (99%) |
| Correct function name | 0/100 (0%) | 99/100 (99%) |
| Exact function name and arguments | 0/100 (0%) | 88/100 (88%) |
| Correct abstention on unrelated prompts | 100/100 (100%) | 27/100 (27%) |
The continuation learned call syntax and argument selection on this test, but
it called the irrelevant available tool on 73 of 100 unrelated prompts.
The parent produced no well-formed calls on the callable prompts. The result
supports improving no-call/relevance training before relying on this model in
an agent that exposes optional tools. Training loss alone did not reveal this
tradeoff.
The test is narrower than full BFCL: it has only 200 examples, the no-call
class uses an artificial unrelated tool, and it does not measure multi-call
planning, tool execution, long-context behavior, or real-world tool choice.
The tokenizer loaders used for the parent and candidate produced identical
input IDs on all 200 evaluation prompts despite a Transformers warning about
the candidate tokenizer regex. LUMI evaluation job 22605103 completed in
44 minutes 23 seconds; the machine-readable results are in the training
workspace at artifacts/eval/tooluse-anneal300b-v2/paired-heldout.json.
The tool-use training data is English-only. The model can produce malformed
arguments, select the wrong function, or call a tool when none is needed.
No safety or multilingual tool-use benchmark has been run for this checkpoint.
License and attribution
The parent model and tool-use dataset are Apache-2.0. The Dolci instruction
replay is ODC-By-1.0; retain its attribution and review the upstream dataset
card and responsible-use terms. Training was performed on LUMI, hosted by CSC
in Finland, for OpenEuroLLM research.
