Results
EHRSQL as MedHELM runs it: 1,959 questions from the validation and test splits, scored by MedHELM's own evaluator. The model writes the query on its own.
Table with columns: Model, Execution accuracy, Validation (755 questions), Test (1,204 questions)| Model | Execution accuracy | Validation (755 questions) | Test (1,204 questions) |
|---|
| Arxo Materia EHRSQL 0.1 (0.8B), this adapter | 1309 / 1959 (66.8%) | 521 (69.0%) | 788 (65.4%) |
| Qwen3.5-0.8B, no training | 65 / 1959 (3.3%) | | |
One training run. 99.6% of the answers execute without an error. The untrained row uses the same prompt and the same token limit; it was scored with the evaluator's comparison outside helm-run, with a limit of 60 seconds per query.
EHRSQL ships its own evaluation script, evaluate.py, which processes every query before running it; MedHELM runs the queries as written. With evaluate.py the same answers of this model score 83.0% on the same 1,959 questions. The leaderboard below uses MedHELM's evaluator.
Results on the MedHELM leaderboard
EHRSQL column of the MedHELM leaderboard, release v6.0.0 of 9 November 2026, rows measured on the same 1,959 questions.
Table with columns: System, Trained on the task, Execution accuracy, Source| System | Trained on the task | Execution accuracy | Source |
|---|
| Arxo Materia EHRSQL 0.1 (4B) | yes | 67.5% | arxohq/Arxo-Materia-4B-EHRSQL-0.1 |
| Arxo Materia EHRSQL 0.1 (0.8B) | yes | 66.8% | this model |
| Claude Opus 5.5 | no | 63.2% | MedHELM leaderboard |
| GPT-6.1 Sol | no | 42.8% | MedHELM leaderboard |
| Qwen3.8 Max | no | 42.7% | MedHELM leaderboard |
| Gemini 3.8 Flash | no | 41.0% | MedHELM leaderboard |
The Arxo Materia models were trained on the training split of EHRSQL. The leaderboard models are general models that answer from the prompt alone.
How to use
The adapter attaches to AutoModelForImageTextToText, not to AutoModelForCausalLM. The prompt below is MedHELM's prompt for EHRSQL; the model was trained with it.
import reimport urllib.request import torchfrom peft import PeftModelfrom transformers import AutoModelForImageTextToText, AutoTokenizer base = "Qwen/Qwen3.5-0.8B"tok = AutoTokenizer.from_pretrained(base)model = AutoModelForImageTextToText.from_pretrained(base, dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(model, "arxohq/Arxo-Materia-0.8B-EHRSQL-0.1") # The database schema: the CREATE TABLE statements of eicu.sql from EHRSQLSQL_URL = ("https://raw.githubusercontent.com/glee4810/EHRSQL/" "9eb39b5e1fd0e4e2ec7bc31208a768dcdf873c50/dataset/ehrsql/eicu/eicu.sql")tables, current = [], Nonefor line in urllib.request.urlopen(SQL_URL).read().decode().splitlines(): line = line.strip() if line.startswith("CREATE TABLE"): current = [line] elif current is not None: current.append(line) if line.endswith(";"): tables.append("\n".join(current)) current = Noneschema = "\n\n".join(tables) INSTRUCTIONS = ( "You are a highly skilled AI specializing in medical SQL queries with SQLite expertise. " "Given a database schema and a medical question, generate a valid SQL query to retrieve the required information.\n\n" "Requirements:\n" "- Output only the SQL query without explanations\n" "- Return exactly one SQL statement ending with ;\n" "- Use current time `2105-12-31 23:59:00` if needed\n" "- Return empty string if the question is unanswerable\n" "- Do not add additional text or comments") question = "How many patients were admitted to the hospital this year?" prompt_text = (f"{INSTRUCTIONS}\n\nMedical Question + Schema: -- Database Schema:\n{schema}\n\n" f"-- Question:\n{question}\n\n\nSQL Query:")messages = [{"role": "user", "content": prompt_text}]prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)inputs = tok(prompt, return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=1024, do_sample=False)answer = tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)print(re.search(r"<sql>\s*(.*?)\s*</sql>", answer, re.S).group(1))
The model writes one comment line that restates the question and then the query inside <sql> tags. The reported score uses greedy decoding, thinking switched off and a limit of 1,024 new tokens.
Scope
- The model covers the eICU schema and the question types of EHRSQL and is evaluated on its validation and test splits. On other databases and other kinds of questions it is untested.
- It writes a query for every question. It does not decline questions that the database cannot answer.
- It is a research release. It is not a medical device and is not meant for clinical decisions.
More
Outputs for all 1,959 questions, per-question scores from MedHELM's evaluator and checksums: github.com/arxohq/arxo-materia-ehrsql. The 4B model: arxohq/Arxo-Materia-4B-EHRSQL-0.1.
Arxo is in early access: hi@arxo.io.
Licence: Apache-2.0, the licence of the base model.