Apertus-EstLLM-8B-Instruct-0326 API & Inference Endpoint

Use with transformers

python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "tartuNLP/Apertus-EstLLM-8B-Instruct-0326"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    dtype="auto",
    device_map="auto"
)

# to use on apple silicon, load the following way
# model = AutoModelForCausalLM.from_pretrained(
#     model_name,
#     dtype=torch.float16,
#     device_map="mps",
# )

tokenizer = AutoTokenizer.from_pretrained(model_name)

messages = [
    {"role": "user", "content": "Kas sa räägid eesti keelt?"}
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

model_inputs = tokenizer(text, return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=128,
    do_sample=True,
    temperature=0.4,
    # specify eos token to stop at the end of the assistant response
    eos_token_id=tokenizer.eos_token_id,
)

# generated_ids include the input tokens as well, so we only decode new tokens
response = tokenizer.decode(
    generated_ids[0][model_inputs["input_ids"].shape[1]:],
    skip_special_tokens=True,
)

print(response)

Evaluation

Logits-based

Scores for logits-based evaluation benchmarks are available on the EuroEval leaderboard.

Generative

Every benchmark in this category is treated as a generative problem, and thus the evaluation is performed on the model responses obtained with 0 temperature (not logits). The top scores are higlighted with bold. Second best scores are highlighted with italic bold. Rows are sorted in descending order based on the number of parameters of models (not scores). The test set is used for evaluation of each dataset unless noted otherwise.

Note that all models are evaluated with the same prompt template for comparability, meaning that the scores do not necessarily represent each model's best possible performance. This is especially the case for deepseek-ai/DeepSeek-V3-0324 on some of the benchmarks.

Only models of comparable size are evaluated on benchmarks in English.

Instruction-following

Estonian

Instruction level strict accuracy is reported for IFEval-et.

Table with columns: Model (# parameters ↓), IFEval-et
Model (# parameters ↓)	IFEval-et
moonshotai/Kimi-K2-Instruct	0.7891
deepseek-ai/DeepSeek-V3.2	0.7221
deepseek-ai/DeepSeek-V3-0324	0.7171
mistralai/Mistral-Large-3-675B-Instruct-2512	0.7097
meta-llama/Llama-3.1-405B-Instruct	0.7159
meta-llama/Llama-3.3-70B-Instruct	*0.7705*
Qwen/Qwen2.5-72B-Instruct	0.7407
google/gemma-3-27b-it

English

Instruction level strict accuracy is reported for IFEval-en.

Table with columns: Model (# parameters ↓), IFEval-en
Model (# parameters ↓)	IFEval-en
utter-project/EuroLLM-9B-Instruct-2512	0.7564
utter-project/EuroLLM-9B-Instruct	0.7004
mistralai/Ministral-3-8B-Instruct-2512	0.6845
tartuNLP/Apertus-EstLLM-8B-Instruct-0326	0.7089
tartuNLP/Apertus-EstLLM-8B-Instruct-1125	0.6638
swiss-ai/Apertus-8B-Instruct-2509	0.7808
meta-llama/Llama-3.1-8B-Instruct	0.8106

Multiple Choice

All datasets except Winogrande-et are evaluated in 0-shot mode. Winogrande-et is evaluated in 3-shot mode. Exact match accuracy is reported for every dataset.

Estonian Language Competence

Table with columns: Model (# parameters ↓), Grammar-et, Inflection-et, Word-Meanings-et
Model (# parameters ↓)	Grammar-et	Inflection-et	Word-Meanings-et
moonshotai/Kimi-K2-Instruct	0.916	0.6458	0.9689
deepseek-ai/DeepSeek-V3.2	0.781	0.6891	0.8134
deepseek-ai/DeepSeek-V3-0324	0.364

Knowledge and Reasoning (Estonian)

Table with columns: Model (# parameters ↓), Winogrande-et, Trivia-et, Exam-et, GlobalPIQA-et, TruthfulQA-et
Model (# parameters ↓)	Winogrande-et	Trivia-et	Exam-et	GlobalPIQA-et	TruthfulQA-et
moonshotai/Kimi-K2-Instruct	0.8138	0.4225

Knowledge and Reasoning (English)

Table with columns: Model (# parameters ↓), Winogrande, GlobalPIQA-en, TruthfulQA, MMLU-Redux, GSM8K
Model (# parameters ↓)	Winogrande	GlobalPIQA-en	TruthfulQA	MMLU-Redux	GSM8K
utter-project/EuroLLM-9B-Instruct-2512	0.5546	0.58	0.4614	0.6334

Translation

English to Estonian

Table with columns: Model, wmt24pp (BLEU ↑)
Model	wmt24pp (BLEU ↑)
BSC-LT/salamandraTA-7b-instruct	0.2713
tartuNLP/Apertus-EstLLM-8B-Instruct-0326	0.2676
tartuNLP/Llama-3.1-EstLLM-8B-Instruct-1125	0.2635
tartuNLP/Llama-3.1-EstLLM-8B-Instruct-0825	0.264
tartuNLP/Apertus-EstLLM-8B-Instruct-1125	0.2609
utter-project/EuroLLM-9B-Instruct	0.2602
utter-project/EuroLLM-9B-Instruct-2512	0.2567
swiss-ai/Apertus-8B-Instruct-2509

Citation

markdown
@misc{dorkin2026estllmenhancingestoniancapabilities,
      title={{EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Training}}, 
      author={Aleksei Dorkin and Taido Purason and Emil Kalbaliyev and Hele-Andra Kuulmets and Marii Ojastu and Mark Fišel and Tanel Alumäe and Eleri Aedmaa and Krister Kruusmaa and Kairit Sirts},
      year={2026},
      eprint={2603.02041},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2603.02041}, 
}

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "tartuNLP/Apertus-EstLLM-8B-Instruct-0326"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    dtype="auto",
    device_map="auto"
)

# to use on apple silicon, load the following way
# model = AutoModelForCausalLM.from_pretrained(
#     model_name,
#     dtype=torch.float16,
#     device_map="mps",
# )

tokenizer = AutoTokenizer.from_pretrained(model_name)

messages = [
    {"role": "user", "content": "Kas sa räägid eesti keelt?"}
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

model_inputs = tokenizer(text, return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=128,
    do_sample=True,
    temperature=0.4,
    # specify eos token to stop at the end of the assistant response
    eos_token_id=tokenizer.eos_token_id,
)

# generated_ids include the input tokens as well, so we only decode new tokens
response = tokenizer.decode(
    generated_ids[0][model_inputs["input_ids"].shape[1]:],
    skip_special_tokens=True,
)

print(response)

Model (# parameters ↓)

IFEval-et

moonshotai/Kimi-K2-Instruct

0.7891

deepseek-ai/DeepSeek-V3.2

0.7221

deepseek-ai/DeepSeek-V3-0324

0.7171

mistralai/Mistral-Large-3-675B-Instruct-2512

0.7097

meta-llama/Llama-3.1-405B-Instruct

0.7159

meta-llama/Llama-3.3-70B-Instruct

0.7705

Qwen/Qwen2.5-72B-Instruct

0.7407

google/gemma-3-27b-it

Model (# parameters ↓)

IFEval-en

utter-project/EuroLLM-9B-Instruct-2512

0.7564

utter-project/EuroLLM-9B-Instruct

0.7004

mistralai/Ministral-3-8B-Instruct-2512

0.6845

tartuNLP/Apertus-EstLLM-8B-Instruct-0326

0.7089

tartuNLP/Apertus-EstLLM-8B-Instruct-1125

0.6638

swiss-ai/Apertus-8B-Instruct-2509

0.7808

meta-llama/Llama-3.1-8B-Instruct

0.8106

Model (# parameters ↓)

Grammar-et

Inflection-et

Word-Meanings-et

moonshotai/Kimi-K2-Instruct

0.916

0.6458

0.9689

deepseek-ai/DeepSeek-V3.2

0.781

0.6891

0.8134

deepseek-ai/DeepSeek-V3-0324

0.364

Table with columns: Model (# parameters ↓), Winogrande-et, Trivia-et, Exam-et, GlobalPIQA-et, TruthfulQA-et
Model (# parameters ↓)	Winogrande-et	Trivia-et	Exam-et	GlobalPIQA-et	TruthfulQA-et
moonshotai/Kimi-K2-Instruct	0.8138	0.4225

Model (# parameters ↓)

moonshotai/Kimi-K2-Instruct

0.8138

0.4225

Table with columns: Model (# parameters ↓), Winogrande, GlobalPIQA-en, TruthfulQA, MMLU-Redux, GSM8K
Model (# parameters ↓)	Winogrande	GlobalPIQA-en	TruthfulQA	MMLU-Redux	GSM8K
utter-project/EuroLLM-9B-Instruct-2512	0.5546	0.58	0.4614	0.6334

Model (# parameters ↓)

utter-project/EuroLLM-9B-Instruct-2512

0.5546

0.58

0.4614

0.6334

Model

wmt24pp (BLEU ↑)

BSC-LT/salamandraTA-7b-instruct

0.2713

tartuNLP/Apertus-EstLLM-8B-Instruct-0326

0.2676

tartuNLP/Llama-3.1-EstLLM-8B-Instruct-1125

0.2635

tartuNLP/Llama-3.1-EstLLM-8B-Instruct-0825

0.264

tartuNLP/Apertus-EstLLM-8B-Instruct-1125

0.2609

utter-project/EuroLLM-9B-Instruct

0.2602

utter-project/EuroLLM-9B-Instruct-2512

0.2567

swiss-ai/Apertus-8B-Instruct-2509

markdown

@misc{dorkin2026estllmenhancingestoniancapabilities,
      title={{EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Training}}, 
      author={Aleksei Dorkin and Taido Purason and Emil Kalbaliyev and Hele-Andra Kuulmets and Marii Ojastu and Mark Fišel and Tanel Alumäe and Eleri Aedmaa and Krister Kruusmaa and Kairit Sirts},
      year={2026},
      eprint={2603.02041},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2603.02041}, 
}

Apertus-EstLLM-8B-Instruct-0326

README

Use with transformers

Evaluation

Logits-based

Generative

Instruction-following

Estonian

English

Multiple Choice

Estonian Language Competence

Knowledge and Reasoning (Estonian)

Knowledge and Reasoning (English)

Translation

English to Estonian

Citation

Explore FriendliAI today

README

Use with transformers

Evaluation

Logits-based

Generative

Instruction-following

Estonian

English

Multiple Choice

Estonian Language Competence

Knowledge and Reasoning (Estonian)

Knowledge and Reasoning (English)

Translation

English to Estonian

Citation