Highlights
- Ophthalmology specialist — Qwen2.5-VL-7B adapted through three stages: pretraining on 1.33 M image–text pairs, supervised fine-tuning on 360 K ophthalmic instructions, and risk-aware DAPO policy optimization.
- Two evaluation axes — multimodal understanding (multiple-choice and short-answer questions) and open-ended clinical generation (report generation and medical consultation).
- Broad coverage — a 22,000-sample internal test set plus four external benchmarks spanning seven imaging modalities and over 300 ophthalmic conditions.
- Leads every free-text benchmark — first on every generation metric across OphthalVQA, FFA-IR and JAMA Challenge against Qwen3-VL, MedGemma and Lingshu (all P < 0.001).
- Clinically reviewed — ranked highest overall for correctness and control of unsupported content in a 50-case blinded review, with 0.94 agreement (95% CI 0.90–0.97) between LLM-judge and ophthalmologist rankings.
- Reproducible out of the box — the shipped
generation_config.json and preprocessor_config.json reproduce the evaluated configuration exactly.
Previous ophthalmic vision–language model (VLM) studies have commonly evaluated visual question
answering and report generation, with limited evidence for medical consultation. Medical
consultation requires models to synthesize clinical context and imaging findings into diagnostic
and management-oriented responses without predefined answers or explicit diagnostic cues. Here we
developed Oculora, a 7B-scale ophthalmic VLM trained on 1.33 million image–text pairs and
360,000 ophthalmic instructions through staged pretraining, supervised fine-tuning and risk-aware
policy optimization. We assessed multimodal understanding with multiple-choice and short-answer
questions and open-ended clinical generation with report generation and medical consultation on a
22,000-sample internal test set and four benchmarks spanning seven modalities and over 300
ophthalmic conditions. Evaluation combined prespecified task-specific metrics with five-point
clinical-quality ratings from large language model judges and senior ophthalmologists.
Multiple-choice accuracy was 0.820 internally and 0.674 on JSIEC. Oculora led all generation
metrics across three external free-text benchmarks (all P < 0.001). On JAMA Challenge, ROUGE-L
was 0.500 versus 0.311 and BERTScore-F1 was 0.927 versus 0.893 for Qwen3-VL-8B. In a 50-case
blinded review, Oculora ranked highest overall for correctness and control of unsupported content;
LLM and ophthalmologist rankings agreed (0.94, 95% CI 0.90–0.97). These findings show that a
specialized ophthalmic VLM can support both multimodal understanding and open-ended clinical
generation.
Quickstart
Requires transformers >= 4.57.1. Greedy decoding and max_pixels = 262144 are already the
shipped defaults, so loading the model as-is reproduces the reported configuration — no extra
flags required.
import torchfrom transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor model = Qwen2_5_VLForConditionalGeneration.from_pretrained( "pxiaoyu/Oculora", dtype=torch.bfloat16, device_map="auto",) # max_pixels = 262144 is already the shipped defaultprocessor = AutoProcessor.from_pretrained("pxiaoyu/Oculora") messages = [{ "role": "user", "content": [ {"type": "image", "url": "https://example.com/fundus.jpg"}, {"type": "text", "text": "Describe the fundus findings and give the most likely diagnosis."}, ],}] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt",).to(model.device) # greedy decoding is already the shipped defaultout = model.generate(**inputs, max_new_tokens=1000)print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
python -m vllm.entrypoints.openai.api_server \ --model pxiaoyu/Oculora \ --served-model-name oculora \ --host 127.0.0.1 --port 8002 \ --dtype bfloat16 \ --max-model-len 8192 \ --gpu-memory-utilization 0.75 \ --max-num-seqs 8 \ --limit-mm-per-prompt '{"image":4}' \ --trust-remote-code
Query it as any chat-completions endpoint with image_url content parts. vLLM reads
generation_config.json from the repo, so "temperature": 0 is already the default.
Hardware — bfloat16 weights are ~17 GB plus KV cache; a single 40 GB A100 or 48 GB L40S is
comfortable at --max-model-len 8192.
Evaluation
Multiple-choice and short-answer questions assess multimodal understanding; report generation
and medical consultation assess open-ended clinical generation. Every model below was evaluated
under an identical protocol on the same test sets.
Table with columns: Multimodal understanding, Oculora, Qwen3-VL, MedGemma, Lingshu| Multimodal understanding | Oculora | Qwen3-VL | MedGemma | Lingshu |
|---|
| Internal MCQ · Accuracy | 0.8197 | 0.3230 | 0.3470 | 0.2993 |
| Internal MCQ · F1 | 0.8303 | 0.3393 | 0.3505 | 0.3020 |
| JSIEC · Accuracy | 0.6740 | 0.1610 | 0.1410 |
Table with columns: Open-ended clinical generation, Oculora, Qwen3-VL, MedGemma, Lingshu| Open-ended clinical generation | Oculora | Qwen3-VL | MedGemma | Lingshu |
|---|
| FFA-IR · BLEU-2 | 0.1485 | 0.0917 | 0.0579 | 0.0110 |
| FFA-IR · ROUGE-L | 0.2222 | 0.1639 | 0.1040 | 0.0829 |
| FFA-IR · METEOR | 0.2393 | 0.1742 | 0.1962 |
Best score per row in bold. Test sets: JSIEC n = 1,000 · OphthalVQA n = 600 ·
FFA-IR n = 766 · JAMA Challenge n = 177, plus a 22,000-sample internal test set.
Baselines are open-weight vision–language models at a comparable parameter scale, run under an
identical protocol on the same test sets: Qwen3-VL-8B,
MedGemma-4B and
Lingshu-7B.
Bootstrap 95% confidence intervals for every value are reported in the paper.
Blinded clinical review
Table | |
|---|
| Design | 50 cases, blinded, ranked by LLM judges and senior ophthalmologists |
| Outcome | Oculora ranked highest overall for correctness and for control of unsupported content |
| Rater agreement | 0.94 (95% CI 0.90 – 0.97) between LLM and ophthalmologist rankings |
Training
Three stages, all on the Qwen2.5-VL-7B backbone.
Table with columns: Stage, Data, Configuration| Stage | Data | Configuration |
|---|
| 1 · Pretraining (PT) | 1.33 M image–text pairs | LoRA, rank 64, α = 128 |
| 2 · Supervised fine-tuning (SFT) | 360 K ophthalmic instructions | LoRA, rank 64, α = 128; vision encoder and multimodal projector frozen |
| 3 · Risk-aware policy optimization | — | Risk-aware DAPO objective in the VERL framework |
Risk-aware DAPO
Table with columns: Setting, Value| Setting | Value |
|---|
| Responses sampled per prompt | n = 8 |
| Global mini-batch per update | 128 prompts → 1,024 responses |
| Risk masking | top-K highest-risk generations (K = 0.1) masked globally across the whole mini-batch, before group-relative advantage computation |
| Reward — keyword | 0.5 |
| Reward — LLM-assessed | 0.4 |
| Reward — format | 0.1 |
Masking the riskiest decile globally rather than within each group is what stabilizes the
gradients: the advantage baseline is never computed over generations the risk model has already
flagged.
Inference settings used for all reported results
Table with columns: Setting, Value| Setting | Value |
|---|
| Decoding | greedy |
| Temperature | 0 |
| Max image pixels | 262,144 |
These are the shipped defaults. preprocessor_config.json sets max_pixels = 262144 and
generation_config.json sets do_sample = false, so loading the model as-is reproduces the
reported configuration — no extra flags required.
Architecture
Table | |
|---|
| Class | Qwen2_5_VLForConditionalGeneration |
| Total parameters | 8.29 B |
| Language model | 28 layers · hidden 3584 · 28 heads · 4 KV heads (GQA) · SwiGLU · RMSNorm |
| Vision encoder | 32 layers · hidden 1280 → 3584 · 16 heads · patch 14 · spatial merge 2 · window 112 |
| Vocabulary | 152,064 |
| Context | 128 K positions · sliding window 32,768 |
| Position encoding | M-RoPE, mrope_section = [16, 24, 24], θ = 1,000,000 |
|
Intended use and limitations
Oculora is a research artifact, intended for research on ophthalmic vision–language modelling
and for retrospective evaluation on de-identified data.
- Not a medical device. Not cleared or approved by any regulatory body. It must not be used
for diagnosis, triage or treatment decisions for real patients.
- Outputs can be fluent and confident while being wrong. Every clinical statement requires review
by a qualified ophthalmologist.
- Performance is characterised on the benchmarks above — seven imaging modalities and 300+
conditions. Behaviour outside that distribution (other modalities, populations, languages, or
image quality) is unmeasured.
- Training data inherits the demographic and disease-prevalence biases of its sources; the model
may underperform on rare conditions and underrepresented groups.
- Do not send protected health information to any hosted inference endpoint.
License
Apache 2.0, inheriting the license of the Qwen2.5-VL base model. Users remain responsible for
complying with the terms of the underlying base model and of every dataset used for evaluation.
Citation
The paper describing Oculora is currently under review; a full citation will be added here upon
publication. Until then, please cite this repository:
pxiaoyu/Oculora.