SweMedMCQ-EuroLLM-1.7B-Trace-EN-SV API & Inference Endpoint

Training

Base model: utter-project/EuroLLM-1.7B-Instruct.
Data: 1,098,916 examples: the English and the Swedish set together (same training questions), from the training splits of the ten MedicalAgentsBench datasets. The Swedish training questions were translated from English with GPT-OSS-120B. No test item was used for training or model selection.
Supervision: Reasoning-trace supervision: responses of two open teacher models to the training questions, kept only when the answer is correct and the explanation is not empty (length outliers removed). GPT-OSS-120B answered each question three times (greedy and twice at temperature 0.7) and its reasoning is kept as a <think>…</think> block before the JSON answer; Qwen3.6-35B-A3B answered twice (greedy and at temperature 0.7) and only its explanation and answer are kept. A question can therefore have up to five training examples. In the Swedish set the teachers answered the Swedish questions; GPT-OSS-120B still reasons mostly in English, while the explanations are in Swedish.
Training language: English and Swedish.
Settings: full fine-tuning (embeddings trained), two epochs planned, effective batch 128 (32 per GPU on 4 GPUs), learning rate 5e-5 with cosine schedule and 5% warm-up, weight decay 0.01, gradient clipping 1.0, bfloat16, padding-free packing, sequences up to 12,288 tokens, loss on the response tokens, seed 13.
Checkpoint: best validation loss, step 8,400 of 9,200, validation loss 0.7676 (evaluated every 200 steps). Early stopping at step 9,200 (1.07 epochs).

Prompt format

{question} is the question text and {options} the options, one per paragraph, as A) …, B) … (letters in order, separated by a blank line). The prompt is sent as a single user message, with no system prompt. The model answers with {"explanation": "...", "answer_index": "<letter>"}, after a <think>…</think> reasoning block.

text

You are a highly knowledgeable and careful medical expert. Your task is to analyze the following multiple-choice medical question and determine the single best answer based on established medical knowledge and reasoning. Follow these steps carefully:1. Read the question and all available answer options.2. Identify key clinical findings, pathophysiology, or reasoning relevant to the question.3. Select the single best answer from the listed options.4. Provide a concise, evidence-based explanation for your choice. --- Return your final output **only** in valid JSON format that matches this schema: {  "explanation": "A brief medical reasoning explaining why this option is correct and others are not.",  "answer_index": "The letter corresponding to the correct answer option (e.g., 'A', 'B', 'C', etc.)."} --- ## Input **Question:**{question} **Options:** {options} ---

text

Du är en mycket kunnig och noggrann medicinsk expert. Din uppgift är att analysera följande flervalsfråga och avgöra vilket alternativ som är det enda bästa svaret baserat på etablerad medicinsk kunskap och resonemang. Följ dessa steg noggrant:1. Läs frågan och alla tillgängliga svarsalternativ.2. Identifiera viktiga kliniska fynd, patofysiologi eller relevant medicinskt resonemang.3. Välj det enda bästa svaret bland de angivna alternativen.4. Ge en kortfattad, evidensbaserad förklaring till ditt val. --- Returnera ditt slutliga svar **endast** i giltigt JSON-format som följer detta schema: {  "explanation": "En kort medicinsk motivering som förklarar varför detta alternativ är korrekt och varför de andra inte är det.",  "answer_index": "Bokstaven som motsvarar det korrekta svarsalternativet (t.ex. 'A', 'B', 'C' osv.)."} --- ## Indata **Fråga:**{question} **Alternativ:** {options} ---

EuroLLM's chat template does not open the thinking block the model was trained with, so it is appended after the generation prompt: <think>\n for reasoning, <think>\n\n</think>\n\n for a direct answer.

Context length. The config keeps EuroLLM's max_position_embeddings of 4,096. Training and evaluation used longer sequences, so load the model with hf_overrides={"max_position_embeddings": 12288} in vLLM (or set it in the config for transformers).

Usage with vLLM

Greedy decoding, as in the evaluation. The two prompts above are in prompts.json in this repository (sv and en); fill them with str.format. Swedish options are listed in label order; English options in their original order.

python

import jsonimport re from huggingface_hub import hf_hub_downloadfrom vllm import LLM, SamplingParamsfrom vllm.sampling_params import StructuredOutputsParams  def answer_letter(text):    """Answer letter in the JSON after the reasoning, or None if the model wrote none."""    match = re.search(r'"answer_index"\s*:\s*"\s*([A-Z])', text.split("</think>")[-1])    return match.group(1) if match else None  MODEL = "Anonymiserad/SweMedMCQ-EuroLLM-1.7B-Trace-EN-SV"PROMPTS = json.load(open(hf_hub_download(MODEL, "prompts.json"), encoding="utf-8"))llm = LLM(MODEL, max_model_len=12288, hf_overrides={"max_position_embeddings": 12288})tok = llm.get_tokenizer() question = "Vilket band av elektromagnetiska vågor används i en termografikamera?"options = {"A": "röntgenstrålning", "B": "mikrovågor", "C": "ultraviolett strålning", "D": "infraröd strålning"}opts = "\n\n".join(f"{k}) {v}" for k, v in sorted(options.items()))user = PROMPTS["sv"].format(question=question, options=opts)messages = [{"role": "user", "content": user}]base = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)prompt = base + "<think>\n"                 # reasoningdirect = base + "<think>\n\n</think>\n\n"   # direct answer, then JSON schema = {"type": "object", "properties": {"explanation": {"type": "string"}, "answer_index": {"type": "string"}}, "required": ["explanation", "answer_index"], "additionalProperties": False}# Reasoning: greedy decoding; the JSON answer follows "</think>".text = llm.generate([prompt], SamplingParams(temperature=0.0, max_tokens=10240))[0].outputs[0].textprint(answer_letter(text)) # Direct answer: JSON-schema decoding right after the empty thinking block.params = SamplingParams(temperature=0.0, max_tokens=10240,                        structured_outputs=StructuredOutputsParams(json=schema))print(json.loads(llm.generate([direct], params)[0].outputs[0].text)["answer_index"]) del llm  # shut the engine down, so the script exits

In the paper's evaluation the JSON after </think> was also produced with schema-constrained decoding (a reasoning parser that switches the constraint on once the block is closed; see the evaluation code in the repository). With unconstrained decoding as above, the model sometimes writes no valid answer (answer_letter returns None); the paper counts such answers as wrong.

Serving with vLLM

Start an OpenAI-compatible server:

bash

vllm serve Anonymiserad/SweMedMCQ-EuroLLM-1.7B-Trace-EN-SV --max-model-len 12288 --hf-overrides '{"max_position_embeddings": 12288}'

and send the same prompt (here messages as in the example above). The server rejects requests whose prompt + max_tokens exceed the context, so ask for at most 10,240 new tokens:

python

from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")# The reasoning block is opened in the prompt, so use the completions endpoint with the templated text.prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False) + "<think>\n"r = client.completions.create(model=MODEL, prompt=prompt, temperature=0.0, max_tokens=10240)print(answer_letter(r.choices[0].text))

tok is the model's tokenizer, as in the example above (or transformers.AutoTokenizer.from_pretrained(MODEL)).

Results

Accuracy (%) on the 9,733 core items of the SweMedMCQ test set (English source and Swedish benchmark text, greedy decoding):

Table with columns: Decoding, English, Swedish, Gap (EN − SV)
Decoding	English	Swedish	Gap (EN − SV)
reasoning	39.4	42.6	-3.2
direct answer	44.2	40.5	3.6

The base EuroLLM-1.7B reaches 17.5% in English and 23.5% in Swedish with the same prompt.

Limitations

Research model. It is not a medical device and must not be used for diagnosis, treatment, or any clinical decision.
Trained and evaluated on exam-style multiple-choice questions only; accuracy is far below that of large models, and explanations can be wrong even when the answer is right.
Swedish training and test text is machine-translated from English and may contain translation errors.

Licence

Apache 2.0, as the base model.

Citation

bibtex

@misc{anonymous2026swemedmcq,  title  = {Benchmarking and Adapting {LLMs} for {English}--{Swedish} Medical Question Answering},  author = {Anonymous},  year   = {2026},  note   = {Under review}}

Training

Base model: utter-project/EuroLLM-1.7B-Instruct.
Data: 1,098,916 examples: the English and the Swedish set together (same training questions), from the training splits of the ten MedicalAgentsBench datasets. The Swedish training questions were translated from English with GPT-OSS-120B. No test item was used for training or model selection.
Supervision: Reasoning-trace supervision: responses of two open teacher models to the training questions, kept only when the answer is correct and the explanation is not empty (length outliers removed). GPT-OSS-120B answered each question three times (greedy and twice at temperature 0.7) and its reasoning is kept as a <think>…</think> block before the JSON answer; Qwen3.6-35B-A3B answered twice (greedy and at temperature 0.7) and only its explanation and answer are kept. A question can therefore have up to five training examples. In the Swedish set the teachers answered the Swedish questions; GPT-OSS-120B still reasons mostly in English, while the explanations are in Swedish.
Training language: English and Swedish.
Settings: full fine-tuning (embeddings trained), two epochs planned, effective batch 128 (32 per GPU on 4 GPUs), learning rate 5e-5 with cosine schedule and 5% warm-up, weight decay 0.01, gradient clipping 1.0, bfloat16, padding-free packing, sequences up to 12,288 tokens, loss on the response tokens, seed 13.
Checkpoint: best validation loss, step 8,400 of 9,200, validation loss 0.7676 (evaluated every 200 steps). Early stopping at step 9,200 (1.07 epochs).

Prompt format

text

You are a highly knowledgeable and careful medical expert.Your task is to analyze the following multiple-choice medical question and determine the single best answer based on established medical knowledge and reasoning.Follow these steps carefully:1. Read the question and all available answer options.2. Identify key clinical findings, pathophysiology, or reasoning relevant to the question.3. Select the single best answer from the listed options.4. Provide a concise, evidence-based explanation for your choice.---Return your final output **only** in valid JSON format that matches this schema:{  "explanation": "A brief medical reasoning explaining why this option is correct and others are not.",  "answer_index": "The letter corresponding to the correct answer option (e.g., 'A', 'B', 'C', etc.)."}---## Input**Question:**{question}**Options:**{options}---

You are a highly knowledgeable and careful medical expert. Your task is to analyze the following multiple-choice medical question and determine the single best answer based on established medical knowledge and reasoning. Follow these steps carefully:1. Read the question and all available answer options.2. Identify key clinical findings, pathophysiology, or reasoning relevant to the question.3. Select the single best answer from the listed options.4. Provide a concise, evidence-based explanation for your choice. --- Return your final output **only** in valid JSON format that matches this schema: {  "explanation": "A brief medical reasoning explaining why this option is correct and others are not.",  "answer_index": "The letter corresponding to the correct answer option (e.g., 'A', 'B', 'C', etc.)."} --- ## Input **Question:**{question} **Options:** {options} ---

text

Du är en mycket kunnig och noggrann medicinsk expert.Din uppgift är att analysera följande flervalsfråga och avgöra vilket alternativ som är det enda bästa svaret baserat på etablerad medicinsk kunskap och resonemang.Följ dessa steg noggrant:1. Läs frågan och alla tillgängliga svarsalternativ.2. Identifiera viktiga kliniska fynd, patofysiologi eller relevant medicinskt resonemang.3. Välj det enda bästa svaret bland de angivna alternativen.4. Ge en kortfattad, evidensbaserad förklaring till ditt val.---Returnera ditt slutliga svar **endast** i giltigt JSON-format som följer detta schema:{  "explanation": "En kort medicinsk motivering som förklarar varför detta alternativ är korrekt och varför de andra inte är det.",  "answer_index": "Bokstaven som motsvarar det korrekta svarsalternativet (t.ex. 'A', 'B', 'C' osv.)."}---## Indata**Fråga:**{question}**Alternativ:**{options}---

Du är en mycket kunnig och noggrann medicinsk expert. Din uppgift är att analysera följande flervalsfråga och avgöra vilket alternativ som är det enda bästa svaret baserat på etablerad medicinsk kunskap och resonemang. Följ dessa steg noggrant:1. Läs frågan och alla tillgängliga svarsalternativ.2. Identifiera viktiga kliniska fynd, patofysiologi eller relevant medicinskt resonemang.3. Välj det enda bästa svaret bland de angivna alternativen.4. Ge en kortfattad, evidensbaserad förklaring till ditt val. --- Returnera ditt slutliga svar **endast** i giltigt JSON-format som följer detta schema: {  "explanation": "En kort medicinsk motivering som förklarar varför detta alternativ är korrekt och varför de andra inte är det.",  "answer_index": "Bokstaven som motsvarar det korrekta svarsalternativet (t.ex. 'A', 'B', 'C' osv.)."} --- ## Indata **Fråga:**{question} **Alternativ:** {options} ---

Usage with vLLM

python

import jsonimport re from huggingface_hub import hf_hub_downloadfrom vllm import LLM, SamplingParamsfrom vllm.sampling_params import StructuredOutputsParams  def answer_letter(text):    """Answer letter in the JSON after the reasoning, or None if the model wrote none."""    match = re.search(r'"answer_index"\s*:\s*"\s*([A-Z])', text.split("</think>")[-1])    return match.group(1) if match else None  MODEL = "Anonymiserad/SweMedMCQ-EuroLLM-1.7B-Trace-EN-SV"PROMPTS = json.load(open(hf_hub_download(MODEL, "prompts.json"), encoding="utf-8"))llm = LLM(MODEL, max_model_len=12288, hf_overrides={"max_position_embeddings": 12288})tok = llm.get_tokenizer() question = "Vilket band av elektromagnetiska vågor används i en termografikamera?"options = {"A": "röntgenstrålning", "B": "mikrovågor", "C": "ultraviolett strålning", "D": "infraröd strålning"}opts = "\n\n".join(f"{k}) {v}" for k, v in sorted(options.items()))user = PROMPTS["sv"].format(question=question, options=opts)messages = [{"role": "user", "content": user}]base = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)prompt = base + "<think>\n"                 # reasoningdirect = base + "<think>\n\n</think>\n\n"   # direct answer, then JSON schema = {"type": "object", "properties": {"explanation": {"type": "string"}, "answer_index": {"type": "string"}}, "required": ["explanation", "answer_index"], "additionalProperties": False}# Reasoning: greedy decoding; the JSON answer follows "</think>".text = llm.generate([prompt], SamplingParams(temperature=0.0, max_tokens=10240))[0].outputs[0].textprint(answer_letter(text)) # Direct answer: JSON-schema decoding right after the empty thinking block.params = SamplingParams(temperature=0.0, max_tokens=10240,                        structured_outputs=StructuredOutputsParams(json=schema))print(json.loads(llm.generate([direct], params)[0].outputs[0].text)["answer_index"]) del llm  # shut the engine down, so the script exits

import jsonimport re from huggingface_hub import hf_hub_downloadfrom vllm import LLM, SamplingParamsfrom vllm.sampling_params import StructuredOutputsParams  def answer_letter(text):    """Answer letter in the JSON after the reasoning, or None if the model wrote none."""    match = re.search(r'"answer_index"\s*:\s*"\s*([A-Z])', text.split("</think>")[-1])    return match.group(1) if match else None  MODEL = "Anonymiserad/SweMedMCQ-EuroLLM-1.7B-Trace-EN-SV"PROMPTS = json.load(open(hf_hub_download(MODEL, "prompts.json"), encoding="utf-8"))llm = LLM(MODEL, max_model_len=12288, hf_overrides={"max_position_embeddings": 12288})tok = llm.get_tokenizer() question = "Vilket band av elektromagnetiska vågor används i en termografikamera?"options = {"A": "röntgenstrålning", "B": "mikrovågor", "C": "ultraviolett strålning", "D": "infraröd strålning"}opts = "\n\n".join(f"{k}) {v}" for k, v in sorted(options.items()))user = PROMPTS["sv"].format(question=question, options=opts)messages = [{"role": "user", "content": user}]base = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)prompt = base + "<think>\n"                 # reasoningdirect = base + "<think>\n\n</think>\n\n"   # direct answer, then JSON schema = {"type": "object", "properties": {"explanation": {"type": "string"}, "answer_index": {"type": "string"}}, "required": ["explanation", "answer_index"], "additionalProperties": False}# Reasoning: greedy decoding; the JSON answer follows "</think>".text = llm.generate([prompt], SamplingParams(temperature=0.0, max_tokens=10240))[0].outputs[0].textprint(answer_letter(text)) # Direct answer: JSON-schema decoding right after the empty thinking block.params = SamplingParams(temperature=0.0, max_tokens=10240,                        structured_outputs=StructuredOutputsParams(json=schema))print(json.loads(llm.generate([direct], params)[0].outputs[0].text)["answer_index"]) del llm  # shut the engine down, so the script exits

Serving with vLLM

Start an OpenAI-compatible server:

bash

vllm serve Anonymiserad/SweMedMCQ-EuroLLM-1.7B-Trace-EN-SV --max-model-len 12288 --hf-overrides '{"max_position_embeddings": 12288}'

vllm serve Anonymiserad/SweMedMCQ-EuroLLM-1.7B-Trace-EN-SV --max-model-len 12288 --hf-overrides '{"max_position_embeddings": 12288}'

and send the same prompt (here messages as in the example above). The server rejects requests whose prompt + max_tokens exceed the context, so ask for at most 10,240 new tokens:

python

from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")# The reasoning block is opened in the prompt, so use the completions endpoint with the templated text.prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False) + "<think>\n"r = client.completions.create(model=MODEL, prompt=prompt, temperature=0.0, max_tokens=10240)print(answer_letter(r.choices[0].text))

from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")# The reasoning block is opened in the prompt, so use the completions endpoint with the templated text.prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False) + "<think>\n"r = client.completions.create(model=MODEL, prompt=prompt, temperature=0.0, max_tokens=10240)print(answer_letter(r.choices[0].text))

tok is the model's tokenizer, as in the example above (or transformers.AutoTokenizer.from_pretrained(MODEL)).

Results

Accuracy (%) on the 9,733 core items of the SweMedMCQ test set (English source and Swedish benchmark text, greedy decoding):

Table with columns: Decoding, English, Swedish, Gap (EN − SV)
Decoding	English	Swedish	Gap (EN − SV)
reasoning	39.4	42.6	-3.2
direct answer	44.2	40.5	3.6

The base EuroLLM-1.7B reaches 17.5% in English and 23.5% in Swedish with the same prompt.

Limitations

Research model. It is not a medical device and must not be used for diagnosis, treatment, or any clinical decision.
Trained and evaluated on exam-style multiple-choice questions only; accuracy is far below that of large models, and explanations can be wrong even when the answer is right.
Swedish training and test text is machine-translated from English and may contain translation errors.

Licence

Apache 2.0, as the base model.

Citation

bibtex

@misc{anonymous2026swemedmcq,  title  = {Benchmarking and Adapting {LLMs} for {English}--{Swedish} Medical Question Answering},  author = {Anonymous},  year   = {2026},  note   = {Under review}}

@misc{anonymous2026swemedmcq,  title  = {Benchmarking and Adapting {LLMs} for {English}--{Swedish} Medical Question Answering},  author = {Anonymous},  year   = {2026},  note   = {Under review}}

SweMedMCQ-EuroLLM-1.7B-Trace-EN-SV

README

Training

Prompt format

Usage with vLLM

Serving with vLLM

Results

Limitations

Licence

Citation

Explore FriendliAI today

README

Training

Prompt format

Usage with vLLM

Serving with vLLM

Results

Limitations

Licence

Citation