from transformers import AutoModelForCausalLM, AutoTokenizerimport torch model_name = "tartuNLP/Apertus-EstLLM-8B-Instruct-1125" model = AutoModelForCausalLM.from_pretrained( model_name, dtype="auto", device_map="auto") # to use on apple silicon, load the following way# model = AutoModelForCausalLM.from_pretrained(# model_name,# dtype=torch.float16,# device_map="mps",# ) tokenizer = AutoTokenizer.from_pretrained(model_name) messages = [ {"role": "user", "content": "Kas sa räägid eesti keelt?"}] text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True) model_inputs = tokenizer(text, return_tensors="pt").to(model.device) generated_ids = model.generate( **model_inputs, max_new_tokens=128, do_sample=True, temperature=0.4, # specify eos token to stop at the end of the assistant response eos_token_id=tokenizer.eos_token_id,) # generated_ids include the input tokens as well, so we only decode new tokensresponse = tokenizer.decode( generated_ids[0][model_inputs["input_ids"].shape[1]:], skip_special_tokens=True,) print(response)
Model Details
Model Description
- Developed by: TartuNLP and TalTechNLP research groups
- Funded by: Estonian Ministry of Education and Research, “Estonian Language Technology Program 2018-2027”
- Model type: Causal Language Model, Instruction-following
- Language(s) (NLP): Estonian, English
- License: Llama 3.1 Community License Agreement
- Finetuned from model: tartuNLP/Llama-3.1-EstLLM-8B-0525
Continued Pre-Training
Continued Pre-Training was performed for a single epoch on:
- Estonian National Corpus (8.6B tokens)
- Python-Edu (3.3B tokens)
- FineMath4-Plus (9.5B tokens)
- General Instruction-Augmented Corpora (7.4B tokens)
- Cosmopedia v2 (6.9B tokens)
Supervised Fine-Tuning
Approximately 764k examples were used for Supervised Fine-Tuning. The examples mainly come from the Tulu 3 SFT mixture and EuroBlocks.
Additional data provided by the Institute of Estonian Language (EKI) was also used. In total about 80% of examples are in English. More details TBA.
Direct Preference Optimization
English-only HelpSteer3 was used as is in the Direct Preference Optimization step, as previous research on Poro 2 models
showed that there's no observable benefit from translating preference pairs.
Evaluation
Logits-based
Scores for logits-based evaluation benchmarks are available on the EuroEval leaderboard.
Generative
Every benchmark in this category is treated as a generative problem, and thus the evaluation is performed on the model responses obtained with 0 temperature (not logits).
The top scores are higlighted with bold. Second best scores are highlighted with italic bold. Rows are sorted in descending order based on the number of parameters of models (not scores).
The test set is used for evaluation of each dataset unless noted otherwise.
Note that all models are evaluated with the same prompt template for comparability, meaning that the scores do not necessarily represent each model's best possible
performance. This is especially the case for deepseek-ai/DeepSeek-V3-0324 on some of the benchmarks.
Only models of comparable size are evaluated on benchmarks in English.
Instruction-following
Estonian
Instruction level strict accuracy is reported for IFEval-et.
Table with columns: Model (# parameters ↓), IFEval-et| Model (# parameters ↓) | IFEval-et |
|---|
| moonshotai/Kimi-K2-Instruct | 0.7891 |
| deepseek-ai/DeepSeek-V3.2 | 0.7221 |
| deepseek-ai/DeepSeek-V3-0324 | 0.7171 |
| mistralai/Mistral-Large-3-675B-Instruct-2512 | 0.7097 |
| meta-llama/Llama-3.1-405B-Instruct | 0.7159 |
| meta-llama/Llama-3.3-70B-Instruct | 0.7705 |
| Qwen/Qwen2.5-72B-Instruct | 0.7407 |
| google/gemma-3-27b-it |
English
Instruction level strict accuracy is reported for IFEval-en.
Table with columns: Model (# parameters ↓), IFEval-en| Model (# parameters ↓) | IFEval-en |
|---|
| utter-project/EuroLLM-9B-Instruct | 0.7004 |
| mistralai/Ministral-3-8B-Instruct-2512 | 0.6845 |
| swiss-ai/Apertus-8B-Instruct-2509 | 0.7808 |
| tartuNLP/Apertus-EstLLM-8B-Instruct-1125 | 0.6638 |
| meta-llama/Llama-3.1-8B-Instruct | 0.8106 |
| tartuNLP/Llama-3.1-EstLLM-8B-Instruct-0825 | 0.7527 |
| tartuNLP/Llammas | 0.4373 |
| BSC-LT/salamandra-7b-instruct |
Multiple Choice
All datasets except Winogrande-et are evaluated in 0-shot mode. Winogrande-et is evaluated in 3-shot mode. Exact match accuracy is reported for every dataset.
Estonian Language Competence
Table with columns: Model (# parameters ↓), Grammar-et, Inflection-et, Word-Meanings-et| Model (# parameters ↓) | Grammar-et | Inflection-et | Word-Meanings-et |
|---|
| moonshotai/Kimi-K2-Instruct | 0.916 | 0.6458 | 0.9689 |
| deepseek-ai/DeepSeek-V3.2 | 0.781 | 0.6891 | 0.8134 |
| deepseek-ai/DeepSeek-V3-0324 | 0.364 |
Knowledge and Reasoning (Estonian)
Knowledge and Reasoning (English)
Translation
English to Estonian
Table with columns: Model, wmt24pp (BLEU ↑)| Model | wmt24pp (BLEU ↑) |
|---|
| BSC-LT/salamandraTA-7b-instruct | 0.2713 |
| tartuNLP/Apertus-EstLLM-8B-Instruct-1125 | 0.2609 |
| tartuNLP/Llama-3.1-EstLLM-8B-Instruct-0825 | 0.264 |
| utter-project/EuroLLM-9B-Instruct | 0.2602 |
| swiss-ai/Apertus-8B-Instruct-2509 | 0.2372 |
| tartuNLP/Llammas | 0.1472 |
| meta-llama/Llama-3.1-8B-Instruct | 0.1406 |
| BSC-LT/salamandra-7b-instruct |
Limitations
We don't recommend using the model for anything other than research purposes due to relatiely small improvements on Estonian and degradation on English.
Citation
@misc{dorkin2026estllmenhancingestoniancapabilities, title={{EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Training}}, author={Aleksei Dorkin and Taido Purason and Emil Kalbaliyev and Hele-Andra Kuulmets and Marii Ojastu and Mark Fišel and Tanel Alumäe and Eleri Aedmaa and Krister Kruusmaa and Kairit Sirts}, year={2026}, eprint={2603.02041}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2603.02041}, }
