thefertilityplan

ivf-bench-qwen9b-vlm-orpo

README

License: apache-2.0

What it is

Base model Qwen/Qwen3.5-9B, fine-tuned with ORPO (Hong et al., 2024) on 500 training pairs built from IVF-Bench. For each case, the chosen response is the highest-scoring frontier model output and the rejected response is the lowest-scoring one, both graded by the same five-rubric judge used in the benchmark.

Table with columns: Setting, Value
Setting	Value
Base	Qwen/Qwen3.5-9B (unified VLM)
Method	ORPO, LoRA r=8 alpha=36, all linear layers including the vision tower
Data	550 preference pairs, 500 used for training and 50 held for evaluation
Epochs	3
Learning rate	7.584e-4
Hardware	2x H100 80GB SXM5, DDP, bf16

Results

Scored by GPT-5.4 against five rubrics on a 1 to 5 scale. The held-out split is 103 cases that were never used for training, hyperparameter search, or judge calibration, so it is the number to trust. Bold marks this model's row for identification only; it does not indicate a best value in any column.

Table with columns: Model, Overall, Morphology, Clinical, Reasoning, Guideline, Recommend, Brier, AUROC
Model	Overall	Morphology	Clinical	Reasoning	Guideline	Recommend	Brier	AUROC
GPT-5.4	4.55	3.98	5.00	4.65	4.23	4.88	0.246	0.562
This model (9B)	4.11

Held-out split, 103 cases. This model is second of eight overall, ahead of Claude Opus 4.6 by 0.10 points (95% CI [0.03, 0.17], p=0.007 by paired bootstrap). Against its own base model it gains 24% overall, with every rubric improving and guideline alignment up 38%.

Read the Brier column carefully. This model's 0.245 is second of the eight behind Sonnet 4.6 at 0.242, and more importantly the whole field loses to a constant predictor set to the cohort's 35% pregnancy rate, which scores 0.227. What post-training fixed is overconfidence, pulling mean stated probability from 0.452 to 0.396 against a true rate of 0.350. That is not predictive skill, and the paper argues the ceiling is set by the available data rather than by the models.

On cost, the two are not billed on the same basis and the honest comparison is per case of inference: Opus 4.6 costs $0.310 p er c a se t h r o ug hi t s A P I, w hi l e t hi s m o d e l o na tw o - GP U in s t an ce a t$ 8.38 per hour and 19 seconds per case costs $0.044, r o ug h l y o n ese v e$ 7.98 figure that appears in our leaderboard for this model is almost entirely judge cost, since its own inference ran on hardware we already operated.

How to run it

The Qwen chat template pre-fills <think> in the generation prompt, so the model emits its reasoning followed by a closing </think> and then the answer. Serve it with a reasoning parser so the two come back separately:

bash
vllm serve thefertilityplan/ivf-bench-qwen9b-vlm-orpo \
    --tensor-parallel-size 2 \
    --max-model-len 16384 \
    --reasoning-parser qwen3

Without --reasoning-parser, the whole generation arrives in message.content and you must split it yourself on </think>. Getting this wrong was a real bug in our own evaluation, described in the paper.

Recommended sampling, from the Qwen 3.5 model card: temperature=1.0, top_p=0.95, top_k=20, repetition_penalty=1.05.

Intended use and limitations

Intended use. Research on clinical reasoning in vision-language models, and as a baseline for IVF-Bench. Nothing more.

Not for clinical use. This model is not approved by any regulator, has not been validated in a clinical trial, and must not inform patient care. The largest randomized trial of AI-guided embryo selection to date did not demonstrate noninferiority to trained embryologists (Illingworth et al., Nature Medicine 2024), which is the standard any clinical claim in this area has to clear.

Known limitations.

Outcome prediction is weak. Held-out AUROC for clinical pregnancy is close to chance. The model writes calibrated-sounding probabilities that do not discriminate well between embryos that implanted and embryos that did not.
Morphology grounding (3.56) and guideline alignment (3.93) are the two weak rubrics for this model, as they are across the benchmark; guideline alignment is in fact the lowest-scoring rubric on average across all eight systems. The judge never sees the embryo image, so the morphology score measures consistency with the Gardner grade supplied in the prompt rather than whether the model read the picture, and an ablation that withholds the image changes morphology grounding by only 0.07 to 0.23 points across the systems tested. Do not read these scores as evidence of image understanding.
Preference pairs came from a single judge (GPT-5.4 produced 92% of the chosen responses), so the model inherits that judge's style and blind spots.
Training data comes from one clinic in Austria. Transfer to other labs, microscopes, and patient populations is untested.
Part of the patient context in training cases is synthetic, sampled from published population distributions rather than measured.

Training data

thefertilityplan/ivf-bench-orpo-qwen9b-clipped, derived from IVF-Bench, which is built on the Kromp et al. (2023) human blastocyst dataset (CC BY 4.0).

Citation

bibtex
@article{correa2026ivfbench,
  author  = {Correa, Andrew G. A. and Yoon, Brittany},
  title   = {{IVF-Bench}: A Rubric-Based Standard for Evaluating
             Vision-Language Models on {IVF} Clinical Reasoning},
  journal = {arXiv preprint},
  year    = {2026}
}

License

Apache 2.0, matching the Qwen 3.5 base model. Embryo images in the underlying dataset remain CC BY 4.0 and must be attributed to Kromp et al. (2023).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

thefertilityplan

Model Tree

Base

Qwen/Qwen3.5-9B

Fine-tuned

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality