ai-mind-lab

CineMR

README

License: apache-2.0

Authors

Kunyang Li1,†, Hai Nguyen1,2,†, Joshua Lowe1,†, Chenguang Zhao3, Peace C. Madueme3, Mehdi Hedjazi Moghari4, Mubarak Shah1,§, Pegah Khosravi1,2,§, Yuzhang Shang1,§

1 Institute for Artificial Intelligence, University of Central Florida 2 Department of Clinical Sciences, College of Medicine, University of Central Florida 3 Nemours Children's Health, Orlando, Florida 4 West Virginia University Medicine Children's Hospital, Morgantown, West Virginia

† Co-first author · § Corresponding author

Model summary

Table

Architecture	`Qwen3VLForConditionalGeneration` (`qwen3_vl`)
Parameters	~8.8B
Precision	bfloat16 (`dtype` in `config.json`)
Base model	`Qwen/Qwen3-VL-8B-Instruct`
SFT init	Merged SFT checkpoint on cardiac VQA
RL algorithm	GRPO (EasyR1), LoRA r=64 / α=128 on language layers (vision frozen during LoRA)
Transformers	Exported with `transformers` 5.8.x

Intended use

Answer questions about cardiac cine / volumetric MRI when given frame images or short video clips.
Supports the structured answer format used in CineMR training: final answers in \boxed{...} and optional <tool_call> blocks for measurement-style reasoning.

Not for clinical decision-making. This model is a research artifact; outputs must not be used for diagnosis or treatment without expert review and appropriate validation.

Table with columns: Artifact, Purpose
Artifact	Purpose
`model.safetensors`	Full merged weights (SFT + GRPO LoRA), single shard
`config.json`	Model architecture and dtype
`tokenizer.json`, `tokenizer_config.json`, `vocab.json`, `merges.txt`, …	Text tokenizer
`preprocessor_config.json`, `video_preprocessor_config.json`	Image / video preprocessing for

Loading

python
import torch
from transformers import AutoModelForVision2Seq, AutoProcessor

repo_id = "ai-mind-lab/CineMR"  # or a local path to this directory

model = AutoModelForVision2Seq.from_pretrained(
    repo_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
processor = AutoProcessor.from_pretrained(repo_id, trust_remote_code=True)

Example: single-image VQA

python
from PIL import Image

image = Image.open("path/to/frame.png").convert("RGB")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": "What is the left ventricular ejection fraction?"},
        ],
    }
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)

with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=2048,
        do_sample=True,
        temperature=0.7,
        repetition_penalty=1.15,
    )

print(processor.decode(out[0], skip_special_tokens=True))

Use the same trust_remote_code=True and bfloat16 settings as in training. For evaluation, match the CineMR prompt template and decoding settings used in your eval script.

Decoding note. Pure greedy decoding (do_sample=False, no repetition penalty) can drive this checkpoint into repetition collapse (a single reasoning sentence repeated until the token cap, with no \boxed{} answer or <tool_call> emitted). The evaluation numbers below were produced with do_sample=True, temperature=0.7, repetition_penalty=1.15, no_repeat_ngram_size=0, max_new_tokens=2048, and 4 sampled rollouts per prompt. Use a repetition penalty (≈1.1–1.2) for stable outputs.

Training procedure (summary)

SFT on CineMR JSONL (train split) starting from Qwen3-VL-8B-Instruct; weights merged to a full transformers checkpoint.
GRPO in EasyR1 with:
- Reward: reward_cardiac_vqa.py (compute_score) — accuracy on \boxed{} answers plus format / tool-use terms.
- Rollout: vLLM, n=2 samples per prompt, max response length 1024.
- Actor LR 1e-5, KL coefficient 0.01, global batch size 4.
- Image frames from preprocessed cine caches; pixel budget aligned with Qwen3-VL (min/max pixels in training config).

LoRA weights are merged into the base checkpoint for Hub deployment.

Evaluation

Evaluated on the CineMR test split (1,191 samples) with the training prompt template, 4 sampled rollouts per prompt (temperature=0.7, repetition_penalty=1.15, max_new_tokens=2048). pass@k is the fraction of items with ≥1 correct rollout; mean rollout acc averages correctness over all rollouts.

Table with columns: Metric, Value
Metric	Value
Mean rollout accuracy	0.378
pass@4 (any correct)	0.553
ROUGE-L	0.620
BERTScore F1	0.974
Ground-truth satisfied	0.370

Accuracy by reasoning layer (pass@4): L1 0.306, L2 0.771, L3 0.906, L4 0.614, L5 0.314, L6 0.563. By clinical stage: phenotype (L1–L4) pass@4 0.572, etiology (L5–L6) pass@4 0.353.

Tool use: tool-decision accuracy 0.891 (precision 0.999), tool recall on required items 99.8% (predicted names ⊆ expected 100%), trace/JSON format validity 99.4%, tool-name set-match 0.862, argument accuracy 0.879.

Numbers are from this GRPO checkpoint evaluated with the project eval_sft pipeline. Re-run on a held-out test split before drawing conclusions; the small GRPO validation set is used only for checkpoint tracking.

Limitations

Trained on public cardiac MRI challenge-style corpora (ACDC, M&Ms, M&Ms-2); generalization to other scanners, sequences, or pathologies is not guaranteed.
GRPO training used a small validation set for checkpoint tracking; prefer held-out test evaluation before drawing conclusions.
Tool-use formatting in outputs may be inconsistent unless prompts and decoding match training.

License

This model inherits terms from Qwen3-VL (Apache 2.0) and your use of CineMR data and any dataset/challenge restrictions (ACDC, M&Ms, etc.). Use only for lawful research purposes.

Citation

If you use CineMR, please cite the base Qwen3-VL model and acknowledge the CineMR dataset and cardiac imaging sources:

bibtex
@misc{cinemr_qwen3vl8b_grpo,
  title        = {CineMR: Augmenting Vision-Language Models with Tool-Integrated Reasoning for Quantitative Cardiac MRI Diagnosis},
  author       = {Li, Kunyang and Nguyen, Hai and Lowe, Joshua and Zhao, Chenguang and Madueme, Peace C. and Moghari, Mehdi Hedjazi and Shah, Mubarak and Khosravi, Pegah and Shang, Yuzhang},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/ai-mind-lab/CineMR}},
  note         = {GRPO checkpoint; dataset at huggingface.co/datasets/ai-mind-lab/CineMR},
}

bibtex
@article{qwen3vl,
  title  = {Qwen3-VL Technical Report},
  author = {Qwen Team},
  year   = {2025},
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

ai-mind-lab

Model Tree

Base

Qwen/Qwen3-VL-8B-Instruct

Fine-tuned

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Authors

Kunyang Li1,†, Hai Nguyen1,2,†, Joshua Lowe1,†, Chenguang Zhao3, Peace C. Madueme3, Mehdi Hedjazi Moghari4, Mubarak Shah1,§, Pegah Khosravi1,2,§, Yuzhang Shang1,§

† Co-first author · § Corresponding author

Model summary

Table

Architecture	`Qwen3VLForConditionalGeneration` (`qwen3_vl`)
Parameters	~8.8B
Precision	bfloat16 (`dtype` in `config.json`)
Base model	`Qwen/Qwen3-VL-8B-Instruct`
SFT init	Merged SFT checkpoint on cardiac VQA
RL algorithm	GRPO (EasyR1), LoRA r=64 / α=128 on language layers (vision frozen during LoRA)
Transformers	Exported with `transformers` 5.8.x

Intended use

Answer questions about cardiac cine / volumetric MRI when given frame images or short video clips.
Supports the structured answer format used in CineMR training: final answers in \boxed{...} and optional <tool_call> blocks for measurement-style reasoning.

Not for clinical decision-making. This model is a research artifact; outputs must not be used for diagnosis or treatment without expert review and appropriate validation.

Table with columns: Artifact, Purpose
Artifact	Purpose
`model.safetensors`	Full merged weights (SFT + GRPO LoRA), single shard
`config.json`	Model architecture and dtype
`tokenizer.json`, `tokenizer_config.json`, `vocab.json`, `merges.txt`, …	Text tokenizer
`preprocessor_config.json`, `video_preprocessor_config.json`	Image / video preprocessing for

Loading

python
import torch
from transformers import AutoModelForVision2Seq, AutoProcessor

repo_id = "ai-mind-lab/CineMR"  # or a local path to this directory

model = AutoModelForVision2Seq.from_pretrained(
    repo_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
processor = AutoProcessor.from_pretrained(repo_id, trust_remote_code=True)

Example: single-image VQA

python
from PIL import Image

image = Image.open("path/to/frame.png").convert("RGB")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": "What is the left ventricular ejection fraction?"},
        ],
    }
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)

with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=2048,
        do_sample=True,
        temperature=0.7,
        repetition_penalty=1.15,
    )

print(processor.decode(out[0], skip_special_tokens=True))

Use the same trust_remote_code=True and bfloat16 settings as in training. For evaluation, match the CineMR prompt template and decoding settings used in your eval script.

Decoding note. Pure greedy decoding (do_sample=False, no repetition penalty) can drive this checkpoint into repetition collapse (a single reasoning sentence repeated until the token cap, with no \boxed{} answer or <tool_call> emitted). The evaluation numbers below were produced with do_sample=True, temperature=0.7, repetition_penalty=1.15, no_repeat_ngram_size=0, max_new_tokens=2048, and 4 sampled rollouts per prompt. Use a repetition penalty (≈1.1–1.2) for stable outputs.

Training procedure (summary)

SFT on CineMR JSONL (train split) starting from Qwen3-VL-8B-Instruct; weights merged to a full transformers checkpoint.
GRPO in EasyR1 with:
- Reward: reward_cardiac_vqa.py (compute_score) — accuracy on \boxed{} answers plus format / tool-use terms.
- Rollout: vLLM, n=2 samples per prompt, max response length 1024.
- Actor LR 1e-5, KL coefficient 0.01, global batch size 4.
- Image frames from preprocessed cine caches; pixel budget aligned with Qwen3-VL (min/max pixels in training config).

LoRA weights are merged into the base checkpoint for Hub deployment.

Evaluation

Table with columns: Metric, Value
Metric	Value
Mean rollout accuracy	0.378
pass@4 (any correct)	0.553
ROUGE-L	0.620
BERTScore F1	0.974
Ground-truth satisfied	0.370

Accuracy by reasoning layer (pass@4): L1 0.306, L2 0.771, L3 0.906, L4 0.614, L5 0.314, L6 0.563. By clinical stage: phenotype (L1–L4) pass@4 0.572, etiology (L5–L6) pass@4 0.353.

Limitations

Trained on public cardiac MRI challenge-style corpora (ACDC, M&Ms, M&Ms-2); generalization to other scanners, sequences, or pathologies is not guaranteed.
GRPO training used a small validation set for checkpoint tracking; prefer held-out test evaluation before drawing conclusions.
Tool-use formatting in outputs may be inconsistent unless prompts and decoding match training.

License

This model inherits terms from Qwen3-VL (Apache 2.0) and your use of CineMR data and any dataset/challenge restrictions (ACDC, M&Ms, etc.). Use only for lawful research purposes.

Citation

If you use CineMR, please cite the base Qwen3-VL model and acknowledge the CineMR dataset and cardiac imaging sources:

bibtex
@misc{cinemr_qwen3vl8b_grpo,
  title        = {CineMR: Augmenting Vision-Language Models with Tool-Integrated Reasoning for Quantitative Cardiac MRI Diagnosis},
  author       = {Li, Kunyang and Nguyen, Hai and Lowe, Joshua and Zhao, Chenguang and Madueme, Peace C. and Moghari, Mehdi Hedjazi and Shah, Mubarak and Khosravi, Pegah and Shang, Yuzhang},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/ai-mind-lab/CineMR}},
  note         = {GRPO checkpoint; dataset at huggingface.co/datasets/ai-mind-lab/CineMR},
}

bibtex
@article{qwen3vl,
  title  = {Qwen3-VL Technical Report},
  author = {Qwen Team},
  year   = {2025},
}

CineMR

README

Authors

Model summary

Intended use

Contents

Loading

Example: single-image VQA

Training procedure (summary)

Evaluation

Limitations

License

Citation

Explore FriendliAI today

README

Authors

Model summary

Intended use

Contents

Loading

Example: single-image VQA

Training procedure (summary)

Evaluation

Limitations

License

Citation