ai-mind-lab

CineMR

README

License: apache-2.0

Authors

Kunyang Li1,†, Hai Nguyen1,2,†, Joshua Lowe1,†, Chenguang Zhao3, Peace C. Madueme3, Mehdi Hedjazi Moghari4, Mubarak Shah1,§, Pegah Khosravi1,2,§, Yuzhang Shang1,§

1 Institute for Artificial Intelligence, University of Central Florida 2 Department of Clinical Sciences, College of Medicine, University of Central Florida 3 Nemours Children's Health, Orlando, Florida 4 West Virginia University Medicine Children's Hospital, Morgantown, West Virginia

† Co-first author · § Corresponding author

Model summary

Table

Architecture	`Qwen3VLForConditionalGeneration` (`qwen3_vl`)
Parameters	~8.8B
Precision	bfloat16 (`dtype` in `config.json`)
Base model	`Qwen/Qwen3-VL-8B-Instruct`
SFT init	Merged SFT checkpoint on cardiac VQA
RL algorithm	GRPO (EasyR1), LoRA r=64 / α=128 on language layers (vision frozen during LoRA)
Transformers	Exported with `transformers` 5.8.x

Intended use

Answer questions about cardiac cine / volumetric MRI when given frame images or short video clips.
Supports the structured answer format used in CineMR training: final answers in \boxed{...} and optional <tool_call> blocks for measurement-style reasoning.

Not for clinical decision-making. This model is a research artifact; outputs must not be used for diagnosis or treatment without expert review and appropriate validation.

Table with columns: Artifact, Purpose
Artifact	Purpose
`model.safetensors`	Full merged weights (SFT + GRPO LoRA), single shard
`config.json`	Model architecture and dtype
`tokenizer.json`, `tokenizer_config.json`, `vocab.json`, `merges.txt`, …	Text tokenizer
`preprocessor_config.json`, `video_preprocessor_config.json`	Image / video preprocessing for

Loading

python
import torch
from transformers import AutoModelForVision2Seq, AutoProcessor

repo_id = "ai-mind-lab/CineMR"  # or a local path to this directory

model = AutoModelForVision2Seq.from_pretrained(
    repo_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
processor = AutoProcessor.from_pretrained(repo_id, trust_remote_code=True)

Example: single-image VQA

python
from PIL import Image

image = Image.open("path/to/frame.png").convert("RGB")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": "What is the left ventricular ejection fraction?"},
        ],
    }
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)

with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=2048,
        do_sample=True,
        temperature=0.7,
        repetition_penalty=1.15,
    )

print(processor.decode(out[0], skip_special_tokens=True))

Decoding note. Pure greedy decoding (do_sample=False, no repetition penalty) can drive this checkpoint into repetition collapse (a single reasoning sentence repeated until the token cap, with no \boxed{} answer or <tool_call> emitted). The evaluation numbers below were produced with do_sample=True, temperature=0.7, repetition_penalty=1.15, no_repeat_ngram_size=0, max_new_tokens=2048, and 4 sampled rollouts per prompt. Use a repetition penalty (≈1.1–1.2) for stable outputs.

Use the same trust_remote_code=True and bfloat16 settings as in training. For evaluation, match the CineMR prompt template and decoding settings used in your eval script.

Training procedure (summary)

SFT on CineMR JSONL (train split) starting from Qwen3-VL-8B-Instruct; weights merged to a full transformers checkpoint.
GRPO in EasyR1 with:
- Reward: reward_cardiac_vqa.py (compute_score) — accuracy on \boxed{} answers plus format / tool-use terms.
- Rollout: vLLM, n=2 samples per prompt, max response length 1024.
- Actor LR 1e-5, KL coefficient 0.01, global batch size 4.
- Image frames from preprocessed cine caches; pixel budget aligned with Qwen3-VL (min/max pixels in training config).

LoRA weights are merged into the base checkpoint for Hub deployment.

Evaluation

Evaluated on the CineMR held-out test split (n=3,320; distinct from the 1,191-sample validation split) with the training prompt template, 4 sampled rollouts per prompt (temperature=0.7, repetition_penalty=1.15, max_new_tokens=2048). pass@4 is the fraction of items with ≥1 correct rollout; mean rollout acc averages correctness over all 4 rollouts (this is the metric reported as "Acc." in the paper). All figures below are the mean ± SD over 3 independent evaluation runs of this exact checkpoint (same weights, stochastic decoding only — this captures rollout-sampling variance, not training-seed variance).

Table with columns: Metric, Value
Metric	Value
Mean rollout accuracy	39.24% ± 0.09pp
pass@4 (any correct)	55.98%
Ground-truth satisfied (pass@1)	38.83%
ROUGE-L	0.620
BERTScore F1	0.974

Accuracy by reasoning layer (mean rollout acc): L1 10.87%, L2 73.22%, L3 66.90%, L4 55.84%, L5 11.46%, L6 37.25%. By clinical stage: L3–4 (clinical-criteria) 65.30% ± 0.66pp, L5–6 (full-diagnosis) 17.02% ± 1.41pp.

Tool use: tool-decision accuracy 89.61% (precision 100%), tool recall on required items 99.99%, redundant tool calls on optional items 0.00%, trace/JSON format validity 99.98%, tool-name set-match 88.28%, argument accuracy 88.28%, predicted names ⊆ expected 100%.

Limitations

Trained on public cardiac MRI challenge-style corpora (ACDC, M&Ms, M&Ms-2); generalization to other scanners, sequences, or pathologies is not guaranteed.
The evaluation above is held-out test-set accuracy (mean ± SD over 3 stochastic-decoding runs of this one checkpoint), which captures rollout-sampling variance but not training-seed variance — true multi-seed (independently trained) variance has not been measured.
A non-VLM control that runs all six computational tools unconditionally and routes the outputs through the same clinical decision tree, with no VLM at all, currently exceeds this checkpoint's own diagnostic accuracy on the same test set. This checkpoint is a proof-of-concept that tool-integrated reasoning is necessary for this task, not evidence that it is the best way to obtain it — see the paper for the full discussion.
Tool-use formatting in outputs may be inconsistent unless prompts and decoding match training.

License

This model inherits terms from Qwen3-VL (Apache 2.0) and your use of CineMR data and any dataset/challenge restrictions (ACDC, M&Ms, etc.). Use only for lawful research purposes.

Citation

If you use CineMR, please cite the base Qwen3-VL model and acknowledge the CineMR dataset and cardiac imaging sources:

bibtex
@misc{cinemr_qwen3vl8b_grpo,
  title        = {CineMR: Augmenting Vision-Language Models with Tool-Integrated Reasoning for Quantitative Cardiac MRI Diagnosis},
  author       = {Li, Kunyang and Nguyen, Hai and Lowe, Joshua and Zhao, Chenguang and Madueme, Peace C. and Moghari, Mehdi Hedjazi and Shah, Mubarak and Khosravi, Pegah and Shang, Yuzhang},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/ai-mind-lab/CineMR}},
  note         = {GRPO checkpoint; dataset at huggingface.co/datasets/ai-mind-lab/CineMR},
}

bibtex
@article{qwen3vl,
  title  = {Qwen3-VL Technical Report},
  author = {Qwen Team},
  year   = {2025},
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

ai-mind-lab

Model Tree

Base

Qwen/Qwen3-VL-8B-Instruct

Fine-tuned

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Authors

Kunyang Li1,†, Hai Nguyen1,2,†, Joshua Lowe1,†, Chenguang Zhao3, Peace C. Madueme3, Mehdi Hedjazi Moghari4, Mubarak Shah1,§, Pegah Khosravi1,2,§, Yuzhang Shang1,§

† Co-first author · § Corresponding author

Model summary

Table

Architecture	`Qwen3VLForConditionalGeneration` (`qwen3_vl`)
Parameters	~8.8B
Precision	bfloat16 (`dtype` in `config.json`)
Base model	`Qwen/Qwen3-VL-8B-Instruct`
SFT init	Merged SFT checkpoint on cardiac VQA
RL algorithm	GRPO (EasyR1), LoRA r=64 / α=128 on language layers (vision frozen during LoRA)
Transformers	Exported with `transformers` 5.8.x

Intended use

Answer questions about cardiac cine / volumetric MRI when given frame images or short video clips.
Supports the structured answer format used in CineMR training: final answers in \boxed{...} and optional <tool_call> blocks for measurement-style reasoning.

Not for clinical decision-making. This model is a research artifact; outputs must not be used for diagnosis or treatment without expert review and appropriate validation.

Table with columns: Artifact, Purpose
Artifact	Purpose
`model.safetensors`	Full merged weights (SFT + GRPO LoRA), single shard
`config.json`	Model architecture and dtype
`tokenizer.json`, `tokenizer_config.json`, `vocab.json`, `merges.txt`, …	Text tokenizer
`preprocessor_config.json`, `video_preprocessor_config.json`	Image / video preprocessing for

Loading

python
import torch
from transformers import AutoModelForVision2Seq, AutoProcessor

repo_id = "ai-mind-lab/CineMR"  # or a local path to this directory

model = AutoModelForVision2Seq.from_pretrained(
    repo_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
processor = AutoProcessor.from_pretrained(repo_id, trust_remote_code=True)

Example: single-image VQA

python
from PIL import Image

image = Image.open("path/to/frame.png").convert("RGB")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": "What is the left ventricular ejection fraction?"},
        ],
    }
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)

with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=2048,
        do_sample=True,
        temperature=0.7,
        repetition_penalty=1.15,
    )

print(processor.decode(out[0], skip_special_tokens=True))

Decoding note. Pure greedy decoding (do_sample=False, no repetition penalty) can drive this checkpoint into repetition collapse (a single reasoning sentence repeated until the token cap, with no \boxed{} answer or <tool_call> emitted). The evaluation numbers below were produced with do_sample=True, temperature=0.7, repetition_penalty=1.15, no_repeat_ngram_size=0, max_new_tokens=2048, and 4 sampled rollouts per prompt. Use a repetition penalty (≈1.1–1.2) for stable outputs.

Use the same trust_remote_code=True and bfloat16 settings as in training. For evaluation, match the CineMR prompt template and decoding settings used in your eval script.

Training procedure (summary)

SFT on CineMR JSONL (train split) starting from Qwen3-VL-8B-Instruct; weights merged to a full transformers checkpoint.
GRPO in EasyR1 with:
- Reward: reward_cardiac_vqa.py (compute_score) — accuracy on \boxed{} answers plus format / tool-use terms.
- Rollout: vLLM, n=2 samples per prompt, max response length 1024.
- Actor LR 1e-5, KL coefficient 0.01, global batch size 4.
- Image frames from preprocessed cine caches; pixel budget aligned with Qwen3-VL (min/max pixels in training config).

LoRA weights are merged into the base checkpoint for Hub deployment.

Evaluation

Table with columns: Metric, Value
Metric	Value
Mean rollout accuracy	39.24% ± 0.09pp
pass@4 (any correct)	55.98%
Ground-truth satisfied (pass@1)	38.83%
ROUGE-L	0.620
BERTScore F1	0.974

Limitations

Trained on public cardiac MRI challenge-style corpora (ACDC, M&Ms, M&Ms-2); generalization to other scanners, sequences, or pathologies is not guaranteed.
The evaluation above is held-out test-set accuracy (mean ± SD over 3 stochastic-decoding runs of this one checkpoint), which captures rollout-sampling variance but not training-seed variance — true multi-seed (independently trained) variance has not been measured.
A non-VLM control that runs all six computational tools unconditionally and routes the outputs through the same clinical decision tree, with no VLM at all, currently exceeds this checkpoint's own diagnostic accuracy on the same test set. This checkpoint is a proof-of-concept that tool-integrated reasoning is necessary for this task, not evidence that it is the best way to obtain it — see the paper for the full discussion.
Tool-use formatting in outputs may be inconsistent unless prompts and decoding match training.

License

This model inherits terms from Qwen3-VL (Apache 2.0) and your use of CineMR data and any dataset/challenge restrictions (ACDC, M&Ms, etc.). Use only for lawful research purposes.

Citation

If you use CineMR, please cite the base Qwen3-VL model and acknowledge the CineMR dataset and cardiac imaging sources:

bibtex
@misc{cinemr_qwen3vl8b_grpo,
  title        = {CineMR: Augmenting Vision-Language Models with Tool-Integrated Reasoning for Quantitative Cardiac MRI Diagnosis},
  author       = {Li, Kunyang and Nguyen, Hai and Lowe, Joshua and Zhao, Chenguang and Madueme, Peace C. and Moghari, Mehdi Hedjazi and Shah, Mubarak and Khosravi, Pegah and Shang, Yuzhang},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/ai-mind-lab/CineMR}},
  note         = {GRPO checkpoint; dataset at huggingface.co/datasets/ai-mind-lab/CineMR},
}

bibtex
@article{qwen3vl,
  title  = {Qwen3-VL Technical Report},
  author = {Qwen Team},
  year   = {2025},
}

CineMR

README

Authors

Model summary

Intended use

Contents

Loading

Example: single-image VQA

Training procedure (summary)

Evaluation

Limitations

License

Citation

Explore FriendliAI today

README

Authors

Model summary

Intended use

Contents

Loading

Example: single-image VQA

Training procedure (summary)

Evaluation

Limitations

License

Citation