sarathi-balakrishnan

Qwen2.5-VL-7B-Medical-LoRA

README

License: apache-2.0

Results at a glance

Table with columns: Configuration, GPU memory (allocated), Avg latency / image, Accuracy (5-image eval)
Configuration	GPU memory (allocated)	Avg latency / image	Accuracy (5-image eval)
FP16 base	15,820 MB	31.10 s	not measured
NF4 base (no adapter)	5,664 MB	15.24 s	83.5%
NF4 + this adapter (merged)	11,336 MB	14.70 s	73.7%

Measured on one NVIDIA DGX Spark (GB10). Latency comes from the 5 benchmark images with torch.cuda.synchronize(). NF4 was faster than FP16 here. That result is specific to this hardware and software stack (GB10, PyTorch nightly), and I didn't check it on other GPUs. I never ran the accuracy eval on FP16, so I can't say how much accuracy NF4 itself costs.

What I'd flag about this experiment

The evaluation is very small. It uses 5 synthetic images that I rendered myself (110 expected terms, 125 expected values), scored with case-insensitive substring matching. A single image changes the average a lot. Treat these numbers as a sanity check, not a benchmark.
The fine-tune hurt structured documents. Three of the five cases scored exactly the same. The whole drop came from the two tabular cases: ICU flowsheet 100% → 72.4%, complex lab panel 82.9% → 61.5%.
Likely causes (hypotheses, not ablated):
- Train/test mismatch: about 912 of the 1,000 training samples were narrative MTSamples transcriptions rendered as plain text images. The eval documents are dense tables and flowsheets.
- Low-resolution training images: they were resized to 256 px max, so small table text is hard to read.
- Truncated targets: responses were capped at 600 characters and sequences at 512 tokens, which teaches short answers.
- No validation during training: I loaded 100 validation samples but disabled evaluation (eval_strategy="no"), so I couldn't see the regression until afterwards.
The LoRA also reached the vision tower. I froze the vision encoder before calling get_peft_model. But the target names (gate_proj, up_proj, down_proj) also match the MLPs in visual.blocks.*, so PEFT added trainable LoRA layers there. The adapter file shows 161,480,704 LoRA parameters in the language model and 28,876,800 in the vision blocks' MLPs. Next time I'd restrict targets with a regex to the language model.
The total parameter count printed during training was wrong. The log reported 4.88B total parameters and "3.9% trainable". bitsandbytes packs two 4-bit weights per byte, so numel() undercounts the quantized layers. Against the real ≈8.3B parameters, the 190.4M LoRA parameters are about 2.3%.
Merging didn't save memory. After merge_and_unload() on the 4-bit model, allocated memory went from 5.7 GB to 11.3 GB. I didn't dig into why. Keeping the adapter unmerged, or merging into a BF16 base and then re-quantizing, are the alternatives to test.
Some quantization paths failed in my environment. AutoAWQ failed because PytorchGELUTanh was removed in transformers 4.57. auto-gptq wouldn't build on this aarch64 / CUDA 12.8 setup. I used bitsandbytes NF4 instead.

What I'd do next: train on real tabular clinical forms at higher image resolution, without truncating targets, with LoRA limited to the language model, a per-epoch validation loss, and a larger held-out eval (hundreds of documents with field-level F1).

Per-case results (base NF4 vs NF4 + adapter)

Table with columns: Case, Base terms, FT terms, Base values, FT values, Base combined, FT combined
Case	Base terms	FT terms	Base values	FT values	Base combined	FT combined
Medication reconciliation	16/20	16/20	17/25	17/25	74.0%	74.0%
ICU flowsheet	19/19	18/19	30/30	15/30	100.0%	72.4%
Cardiology consult	20/28	20/28	18/26	18/26	70.3%	70.3%
Complex lab panel	18/22	13/22	21/25	16/25	82.9%	61.5%
Surgical operative note	18/21	18/21	18/19	18/19	90.2%	90.2%
Average	83.79%	78.19%	83.19%	69.19%	83.49%	73.70% (−9.79 pp)

Combined accuracy = (term accuracy + value accuracy) / 2. Matching is case-insensitive substring.

Where the fine-tuned model failed: in the ICU flowsheet it missed 15 numeric values, including FiO2, PEEP, MAP, CVP, pH, pCO2, and platelets. In the lab panel it missed ALT, AST, ALP, GGT, and LDH.

Memory and latency (all configurations)

Table with columns: Configuration, Allocated (MB), Reserved (MB), Peak allocated (MB)
Configuration	Allocated (MB)	Reserved (MB)	Peak allocated (MB)
FP16 base (post-load)	15,820	17,120	16,860
NF4 base (post-load)	5,664	7,018	6,704
NF4 base (eval peak)	5,671	7,508	7,388
NF4 + adapter merged	11,336	14,806	12,894
Training peak	~18,300	—	—

Table with columns: Image, FP16, NF4 base, NF4 + adapter
Image	FP16	NF4 base	NF4 + adapter
Patient diagnosis	28.41 s	15.40 s	13.24 s
Radiology report	22.58 s	11.76 s	12.01 s
Prescription	23.83 s	13.67 s	12.75 s
Lab results	48.74 s	23.25 s	23.46 s
Discharge summary	31.96 s	12.13 s	12.05 s
Average	31.10 s	15.24 s	14.70 s

Load time: FP16 88.7 s, NF4 92.7 s, NF4 + merge 92.7 s.

Training details

Table with columns: Parameter, Value
Parameter	Value
Quantization during training	bitsandbytes NF4, double quantization, BF16 compute (QLoRA)
LoRA	r = 64, α = 128, dropout 0.05, bias none
Target names	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj (matched in the language model and vision MLPs; see above)
Trainable parameters	190,357,504 (161.5M language model + 28.9M vision MLPs), ≈2.3% of ≈8.3B
Adapter file	≈727 MiB (FP32)
Training samples	1,000, subsampled (seed 42) from a 3,916-sample train split: ~912 MTSamples rendered as images with PIL, ~88 PathVQA
Excluded	PubMedVision (all answers under 50 characters after filtering)
Epochs / steps	2 epochs, 124 optimizer steps
Batch	4 per device × 4 grad accumulation = 16
Learning rate	2e-4, cosine, warmup 0.05, weight decay 0.01, max grad norm 1.0
Sequence / image	512 tokens max, images resized to 256 px max side, responses truncated to 600 characters
Loss	assistant tokens only
Evaluation during training	disabled
Time / memory	7,704 s (128.4 min, ~62 s/step), peak ~18.3 GB

Loss: 1.754 at step 5, then 0.730 at step 120 (last logged step). The average training loss over the run was 0.996.

Table with columns: Step, 5, 20, 40, 60, 80, 100, 120
Step	5	20	40	60	80	100	120
Loss	1.754	1.271	1.119	1.037	0.767	0.809	0.730

Environment

GB10 is compute capability 12.1 (sm_121). At training time that needed a PyTorch nightly (2.11.0.dev20260206+cu128) plus a small compatibility patch that routes integer prod / cumprod to the CPU. Other versions: transformers 4.57.6, PEFT 0.18.0, bitsandbytes 0.49.1, accelerate 1.12.0, datasets 4.4.2, Python 3.10.19, CUDA 12.8.

Usage

Recommended: base model in NF4 (no adapter)

python

import torchfrom transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration, BitsAndBytesConfigfrom qwen_vl_utils import process_vision_infofrom PIL import Image bnb_config = BitsAndBytesConfig(    load_in_4bit=True,    bnb_4bit_quant_type="nf4",    bnb_4bit_compute_dtype=torch.bfloat16,    bnb_4bit_use_double_quant=True,)model = Qwen2_5_VLForConditionalGeneration.from_pretrained(    "Qwen/Qwen2.5-VL-7B-Instruct", quantization_config=bnb_config, device_map="auto",).eval()processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct") image = Image.open("medical_document.png")messages = [{    "role": "user",    "content": [        {"type": "image", "image": image},        {"type": "text", "text": "Extract all medical diagnoses, medications with dosages, and vital signs from this document."},    ],}]text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)image_inputs, video_inputs = process_vision_info(messages)inputs = processor(text=[text], images=image_inputs, videos=video_inputs,                   padding=True, return_tensors="pt").to(model.device) with torch.no_grad():    output_ids = model.generate(**inputs, max_new_tokens=1024)print(processor.decode(output_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

To reproduce the fine-tuned numbers: add this adapter

python

from peft import PeftModelmodel = PeftModel.from_pretrained(model, "sarathi-balakrishnan/Qwen2.5-VL-7B-Medical-LoRA")model = model.merge_and_unload().eval()   # this is how the 73.7% / 11.3 GB numbers were measured

Example prompts used in testing:

"Extract all lab values with their results, units, reference ranges, and flags from this laboratory report."
"Extract all ventilator settings, hemodynamic parameters, drip rates, lab values, intake/output totals, and diagnoses from this ICU flowsheet."
"Extract all prescribed medications with their dosages, directions, quantities, and refill counts from this prescription."

Reference API (GitHub repo)

The GitHub repo includes a small FastAPI server (/health, /extract, /analyze, /benchmark) and a Dockerfile. It passed my 5 smoke tests on the GB10. It is a demo, not a hardened service. Note: the server currently loads this adapter. Given the results above, switch it to the base NF4 model.

Intended use and limitations

Research and education only. Not for clinical decision-making, diagnosis, or patient-facing use. Every output needs review by a qualified person.
Evaluated only on 5 synthetic, cleanly rendered English documents. Not validated on real scans, handwriting, stamps, or noisy images.
Can hallucinate or omit terms and values. The fine-tuned version does this more often than the base model on tabular documents.
English only; US clinical abbreviations.

License

Apache-2.0, same as the base model. The training datasets carry their own licenses.

Citation

bibtex

@misc{qwen25vl-medical-lora-2026,  title  = {Qwen2.5-VL-7B-Medical-LoRA: a QLoRA experiment on medical document extraction (negative result)},  author = {Sarathi Balakrishnan},  year   = {2026},  url    = {https://huggingface.co/sarathi-balakrishnan/Qwen2.5-VL-7B-Medical-LoRA}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

sarathi-balakrishnan

Model Tree

Base

Qwen/Qwen2.5-VL-7B-Instruct

Adapter

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Results at a glance

Table with columns: Configuration, GPU memory (allocated), Avg latency / image, Accuracy (5-image eval)
Configuration	GPU memory (allocated)	Avg latency / image	Accuracy (5-image eval)
FP16 base	15,820 MB	31.10 s	not measured
NF4 base (no adapter)	5,664 MB	15.24 s	83.5%
NF4 + this adapter (merged)	11,336 MB	14.70 s	73.7%

What I'd flag about this experiment

The evaluation is very small. It uses 5 synthetic images that I rendered myself (110 expected terms, 125 expected values), scored with case-insensitive substring matching. A single image changes the average a lot. Treat these numbers as a sanity check, not a benchmark.
The fine-tune hurt structured documents. Three of the five cases scored exactly the same. The whole drop came from the two tabular cases: ICU flowsheet 100% → 72.4%, complex lab panel 82.9% → 61.5%.
Likely causes (hypotheses, not ablated):
- Train/test mismatch: about 912 of the 1,000 training samples were narrative MTSamples transcriptions rendered as plain text images. The eval documents are dense tables and flowsheets.
- Low-resolution training images: they were resized to 256 px max, so small table text is hard to read.
- Truncated targets: responses were capped at 600 characters and sequences at 512 tokens, which teaches short answers.
- No validation during training: I loaded 100 validation samples but disabled evaluation (eval_strategy="no"), so I couldn't see the regression until afterwards.
The LoRA also reached the vision tower. I froze the vision encoder before calling get_peft_model. But the target names (gate_proj, up_proj, down_proj) also match the MLPs in visual.blocks.*, so PEFT added trainable LoRA layers there. The adapter file shows 161,480,704 LoRA parameters in the language model and 28,876,800 in the vision blocks' MLPs. Next time I'd restrict targets with a regex to the language model.
The total parameter count printed during training was wrong. The log reported 4.88B total parameters and "3.9% trainable". bitsandbytes packs two 4-bit weights per byte, so numel() undercounts the quantized layers. Against the real ≈8.3B parameters, the 190.4M LoRA parameters are about 2.3%.
Merging didn't save memory. After merge_and_unload() on the 4-bit model, allocated memory went from 5.7 GB to 11.3 GB. I didn't dig into why. Keeping the adapter unmerged, or merging into a BF16 base and then re-quantizing, are the alternatives to test.
Some quantization paths failed in my environment. AutoAWQ failed because PytorchGELUTanh was removed in transformers 4.57. auto-gptq wouldn't build on this aarch64 / CUDA 12.8 setup. I used bitsandbytes NF4 instead.

Per-case results (base NF4 vs NF4 + adapter)

Table with columns: Case, Base terms, FT terms, Base values, FT values, Base combined, FT combined
Case	Base terms	FT terms	Base values	FT values	Base combined	FT combined
Medication reconciliation	16/20	16/20	17/25	17/25	74.0%	74.0%
ICU flowsheet	19/19	18/19	30/30	15/30	100.0%	72.4%
Cardiology consult	20/28	20/28	18/26	18/26	70.3%	70.3%
Complex lab panel	18/22	13/22	21/25	16/25	82.9%	61.5%
Surgical operative note	18/21	18/21	18/19	18/19	90.2%	90.2%
Average	83.79%	78.19%	83.19%	69.19%	83.49%	73.70% (−9.79 pp)

Combined accuracy = (term accuracy + value accuracy) / 2. Matching is case-insensitive substring.

Where the fine-tuned model failed: in the ICU flowsheet it missed 15 numeric values, including FiO2, PEEP, MAP, CVP, pH, pCO2, and platelets. In the lab panel it missed ALT, AST, ALP, GGT, and LDH.

Memory and latency (all configurations)

Table with columns: Configuration, Allocated (MB), Reserved (MB), Peak allocated (MB)
Configuration	Allocated (MB)	Reserved (MB)	Peak allocated (MB)
FP16 base (post-load)	15,820	17,120	16,860
NF4 base (post-load)	5,664	7,018	6,704
NF4 base (eval peak)	5,671	7,508	7,388
NF4 + adapter merged	11,336	14,806	12,894
Training peak	~18,300	—	—

Table with columns: Image, FP16, NF4 base, NF4 + adapter
Image	FP16	NF4 base	NF4 + adapter
Patient diagnosis	28.41 s	15.40 s	13.24 s
Radiology report	22.58 s	11.76 s	12.01 s
Prescription	23.83 s	13.67 s	12.75 s
Lab results	48.74 s	23.25 s	23.46 s
Discharge summary	31.96 s	12.13 s	12.05 s
Average	31.10 s	15.24 s	14.70 s

Load time: FP16 88.7 s, NF4 92.7 s, NF4 + merge 92.7 s.

Training details

Table with columns: Parameter, Value
Parameter	Value
Quantization during training	bitsandbytes NF4, double quantization, BF16 compute (QLoRA)
LoRA	r = 64, α = 128, dropout 0.05, bias none
Target names	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj (matched in the language model and vision MLPs; see above)
Trainable parameters	190,357,504 (161.5M language model + 28.9M vision MLPs), ≈2.3% of ≈8.3B
Adapter file	≈727 MiB (FP32)
Training samples	1,000, subsampled (seed 42) from a 3,916-sample train split: ~912 MTSamples rendered as images with PIL, ~88 PathVQA
Excluded	PubMedVision (all answers under 50 characters after filtering)
Epochs / steps	2 epochs, 124 optimizer steps
Batch	4 per device × 4 grad accumulation = 16
Learning rate	2e-4, cosine, warmup 0.05, weight decay 0.01, max grad norm 1.0
Sequence / image	512 tokens max, images resized to 256 px max side, responses truncated to 600 characters
Loss	assistant tokens only
Evaluation during training	disabled
Time / memory	7,704 s (128.4 min, ~62 s/step), peak ~18.3 GB

Loss: 1.754 at step 5, then 0.730 at step 120 (last logged step). The average training loss over the run was 0.996.

Table with columns: Step, 5, 20, 40, 60, 80, 100, 120
Step	5	20	40	60	80	100	120
Loss	1.754	1.271	1.119	1.037	0.767	0.809	0.730

Environment

Usage

Recommended: base model in NF4 (no adapter)

python

import torchfrom transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration, BitsAndBytesConfigfrom qwen_vl_utils import process_vision_infofrom PIL import Image bnb_config = BitsAndBytesConfig(    load_in_4bit=True,    bnb_4bit_quant_type="nf4",    bnb_4bit_compute_dtype=torch.bfloat16,    bnb_4bit_use_double_quant=True,)model = Qwen2_5_VLForConditionalGeneration.from_pretrained(    "Qwen/Qwen2.5-VL-7B-Instruct", quantization_config=bnb_config, device_map="auto",).eval()processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct") image = Image.open("medical_document.png")messages = [{    "role": "user",    "content": [        {"type": "image", "image": image},        {"type": "text", "text": "Extract all medical diagnoses, medications with dosages, and vital signs from this document."},    ],}]text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)image_inputs, video_inputs = process_vision_info(messages)inputs = processor(text=[text], images=image_inputs, videos=video_inputs,                   padding=True, return_tensors="pt").to(model.device) with torch.no_grad():    output_ids = model.generate(**inputs, max_new_tokens=1024)print(processor.decode(output_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

import torchfrom transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration, BitsAndBytesConfigfrom qwen_vl_utils import process_vision_infofrom PIL import Image bnb_config = BitsAndBytesConfig(    load_in_4bit=True,    bnb_4bit_quant_type="nf4",    bnb_4bit_compute_dtype=torch.bfloat16,    bnb_4bit_use_double_quant=True,)model = Qwen2_5_VLForConditionalGeneration.from_pretrained(    "Qwen/Qwen2.5-VL-7B-Instruct", quantization_config=bnb_config, device_map="auto",).eval()processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct") image = Image.open("medical_document.png")messages = [{    "role": "user",    "content": [        {"type": "image", "image": image},        {"type": "text", "text": "Extract all medical diagnoses, medications with dosages, and vital signs from this document."},    ],}]text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)image_inputs, video_inputs = process_vision_info(messages)inputs = processor(text=[text], images=image_inputs, videos=video_inputs,                   padding=True, return_tensors="pt").to(model.device) with torch.no_grad():    output_ids = model.generate(**inputs, max_new_tokens=1024)print(processor.decode(output_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

To reproduce the fine-tuned numbers: add this adapter

python

from peft import PeftModelmodel = PeftModel.from_pretrained(model, "sarathi-balakrishnan/Qwen2.5-VL-7B-Medical-LoRA")model = model.merge_and_unload().eval()   # this is how the 73.7% / 11.3 GB numbers were measured

from peft import PeftModelmodel = PeftModel.from_pretrained(model, "sarathi-balakrishnan/Qwen2.5-VL-7B-Medical-LoRA")model = model.merge_and_unload().eval()   # this is how the 73.7% / 11.3 GB numbers were measured

Example prompts used in testing:

"Extract all lab values with their results, units, reference ranges, and flags from this laboratory report."
"Extract all ventilator settings, hemodynamic parameters, drip rates, lab values, intake/output totals, and diagnoses from this ICU flowsheet."
"Extract all prescribed medications with their dosages, directions, quantities, and refill counts from this prescription."

Reference API (GitHub repo)

Intended use and limitations

Research and education only. Not for clinical decision-making, diagnosis, or patient-facing use. Every output needs review by a qualified person.
Evaluated only on 5 synthetic, cleanly rendered English documents. Not validated on real scans, handwriting, stamps, or noisy images.
Can hallucinate or omit terms and values. The fine-tuned version does this more often than the base model on tabular documents.
English only; US clinical abbreviations.

License

Apache-2.0, same as the base model. The training datasets carry their own licenses.

Citation

bibtex

@misc{qwen25vl-medical-lora-2026,  title  = {Qwen2.5-VL-7B-Medical-LoRA: a QLoRA experiment on medical document extraction (negative result)},  author = {Sarathi Balakrishnan},  year   = {2026},  url    = {https://huggingface.co/sarathi-balakrishnan/Qwen2.5-VL-7B-Medical-LoRA}}

@misc{qwen25vl-medical-lora-2026,  title  = {Qwen2.5-VL-7B-Medical-LoRA: a QLoRA experiment on medical document extraction (negative result)},  author = {Sarathi Balakrishnan},  year   = {2026},  url    = {https://huggingface.co/sarathi-balakrishnan/Qwen2.5-VL-7B-Medical-LoRA}}