Results
MedCalc-Bench Verified, 1,100 test cases, the benchmark's own evaluator. The model answers alone, without tools.
Table with columns: Model, Accuracy, Formulas (720 cases), Scoring scales (380 cases)| Model | Accuracy | Formulas (720 cases) | Scoring scales (380 cases) |
|---|
| Arxo Materia 0.1 (4B), this adapter | 903 / 1100 (82.1%) | 662 (91.9%) | 241 (63.4%) |
| Qwen3.5-4B, no training, one example of the same calculator in the prompt | 723 / 1100 (65.7%) | 574 (79.7%) | 149 (39.2%) |
Across three training runs the model scores 900, 909 and 903 of 1,100: 82.2 ± 0.4%. This adapter is the run closest to the mean.
Paired with Arxo, which computes the answer from the facts the model reads in the note, Arxo Materia 0.1 reaches 90.2% on the same test set. That pairing is in early access: arxo.io/research/materia-0-1.
Published results on MedCalc-Bench Verified
Systems that take the clinical note and the question as input.
Table with columns: System, Trained on the task, Accuracy, Source| System | Trained on the task | Accuracy | Source |
|---|
| Arxo Materia 0.1 + Arxo | yes | 90.2% | arxo.io/research/materia-0-1; Arxo is in early access |
| Qwen3-8B extracts inputs, calculators written by GPT-5.5 | yes | 86.73% | arXiv 2610.21477, Table II |
| Claude Sonnet 4.6, code agent with Python | no | 86.24% | reported in arXiv 2610.21477 |
| Arxo Materia 0.1 (4B), no tools | yes | 82.2% | this model; mean of three training runs |
| Nemotron 3.5 Super VL (120B) + LoRA, direct answer | yes | 72.45% | Fastino model card |
| Arxo Materia 0.1 (0.8B), no tools | yes | 70.9% | arxohq/Arxo-Materia-0.8B-MedCalc-0.1 |
| Qwen3-8B, supervised fine-tuning, direct answer | yes | 69.46% | arXiv 2610.21477, Table II |
| Qwen3-8B, reinforcement learning, direct answer | yes | 64.2% | arXiv 2512.19691, Appendix E; December 2025 snapshot of Verified |
| Qwen3-8B, one example in the prompt | no | 62.73% | arXiv 2610.21477, Table II |
| GLM-4.6V, direct answer | no | 51.9% | arXiv 2603.02222, Table 3 |
| Qwen2.5-32B, direct answer from the note | no | 28.67% | arXiv 2609.10728, Table 1; first 120 words of the note |
How to use
The adapter attaches to AutoModelForImageTextToText, not to AutoModelForCausalLM. Use the system prompt below; the model was trained with it.
import torchfrom transformers import AutoModelForImageTextToText, AutoTokenizerfrom peft import PeftModel base = "Qwen/Qwen3.5-4B"tok = AutoTokenizer.from_pretrained(base)model = AutoModelForImageTextToText.from_pretrained(base, dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(model, "arxohq/Arxo-Materia-4B-MedCalc-0.1") SYSTEM = ("You are a clinician solving a medical calculation task from a patient note. " "Write the solution step by step: the values taken from the note, unit conversions, each formula with the numbers substituted and worked out one operation at a time; for a score, one line per item with the supporting quote from the note or `not mentioned`, the decision and the points. " 'Finish with one line {"answer": "<value>"}.') note = "..." # the clinical notequestion = "..." # for example: What is the patient's Creatinine Clearance using the Cockcroft-Gault Equation in terms of mL/min? messages = [ {"role": "system", "content": SYSTEM}, {"role": "user", "content": f"Patient note:\n{note}\n\nQuestion: {question}"},]prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)inputs = tok(prompt, return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=2500, do_sample=False)print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
The last line of the output is {"answer": "<value>"}. The reported scores use greedy decoding, thinking switched off and a limit of 2,500 new tokens.
Scope
- The model covers the 55 calculators of MedCalc-Bench and is evaluated on its test split. On other calculators and on notes of a different kind it is untested.
- It is a research release. It is not a medical device and is not meant for clinical decisions.
- Scoring scales are harder for it than formulas, as the table shows.
More
Outputs for all 1,100 test cases, per-case scores for the three runs and checksums: github.com/arxohq/arxo-medcalc-bench.
Licence: Apache-2.0, the licence of the base model.