qwen3-vl-2b-cataract-sft API & Inference Endpoint

Model & Data

Table

Base	`Qwen/Qwen3-VL-2B-Instruct` (`qwen3_vl`, patch 16, 2B)
Dataset	`shahedm2001/dataset_sft` — cataract surgery video, LLaVA format (`video` + `conversations`)
Train / Val	5,597 / 689 samples (2,174 train videos: 2,066 clips + 108 full procedures)
Task	Describe scene + chain-of-thought MCQ (step / visual observation / instrument), teacher/critic descriptions, timestamped full-video narration
Quantization	QLoRA 4-bit during training (NF4, double_quant, `bnb_4bit_compute_dtype=bf16`); merged output is full-precision bf16

Training Configuration

Table with columns: Group, Argument, Value
Group	Argument	Value
Model	`model_id`	`Qwen/Qwen3-VL-2B-Instruct`
	`bits` / `quant_type`	4 / `nf4` (`double_quant True`)
LoRA	`lora_rank` / `lora_alpha` /

bash
.venv/bin/python -u src/train/train_sft.py --model_id Qwen/Qwen3-VL-2B-Instruct \
  --data_path data/sft_train_dataset_sft.json --eval_path data/sft_val_dataset_sft.json \
  --output_dir output/sft_lora --bits 4 --lora_enable True --vision_lora True --use_dora False \
  --lora_rank 16 --lora_alpha 32 --lora_dropout 0.05 --num_lora_modules -1 \
  --lora_namespan_exclude "['lm_head','embed_tokens','merger','pos_embed']" \
  --freeze_vision_tower True --freeze_llm True --freeze_merger False \
  --bf16 True --fp16 False --tf32 True --disable_flash_attn2 True --use_liger_kernel True \
  --num_train_epochs 2 --per_device_train_batch_size 4 --gradient_accumulation_steps 4 \
  --learning_rate 1e-4 --vision_lr 2e-6 --merger_lr 1e-5 --weight_decay 0.1 --warmup_steps 10 \
  --lr_scheduler_type cosine --video_min_pixels 98304 --video_max_pixels 196608 --nframes 64 \
  --max_seq_length 32768 --gradient_checkpointing True --image_folder dataset_sft

Training Curves

Training curves

Table with columns: Metric, First, Last, Best
Metric	First	Last	Best
loss	2.166	1.062	0.6804
grad_norm	2.50 (→ peaks ~33 at step 700, cosine decay tail)
learning_rate	`5.2e-10` at step 700 (cosine schedule, 2 epochs)

700 log lines (output/logs/sft/losses.csv); wall time ≈ 11h on 1× 49 GB GPU (49140 MiB, sdpa attention, bf16).

Environment & Reproducibility

bash
git clone https://github.com/shahedmomenzadeh/qwen3-VL-2B-finetune.git
cd qwen3-VL-2B-finetune
bash setup.sh  # uv + .venv (python 3.12)
source .venv/bin/activate
export HF_HOME=$PWD/hf_cache PYTHONPATH=src

Table with columns: Package, Version
Package	Version
torch	2.14.0+cu130
transformers	5.17.0.dev0
peft	0.19.1
trl	1.12.0
accelerate	1.14.0
bitsandbytes	0.49.2
qwen-vl-utils	0.0.14
liger-kernel	0.8.0

Inference — Control `fps` / `nframes` and Resolution

This is a standard transformers + qwen_vl_utils checkpoint. No quantization needed.

python
import torch
from transformers import AutoProcessor, AutoModelForImageTextToText
from qwen_vl_utils import process_vision_info

model_id = "shahedm2001/qwen3-vl-2b-cataract-sft"  # <-- update after publish
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    model_id, dtype=torch.bfloat16, device_map="auto",
    attn_implementation="sdpa", trust_remote_code=True)
model.eval()

video = "path/to/clip.mp4"
question = "Describe what is happening in this cataract surgical video clip."

# --- choose ONE of nframes / fps, plus desired pixel bounds ---
messages = [{"role": "user", "content": [
    {"type": "video", "video": video,
     "nframes": 64,                 # ← max frames; use one of nframes/fps
     # "fps": 1.0,                  # ← alternative: frames per second
     "min_pixels": 98_304,          # 96*32*32 — lower bound
     "max_pixels": 196_608},        # 192*32*32 — upper bound (increase for more detail / more tokens)
    {"type": "text", "text": question}
]}]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs, video_kwargs = process_vision_info(
    messages, return_video_kwargs=True)
video_kwargs = {k: v for k, v in (video_kwargs or {}).items() if k != "fps"}
inputs = processor(text=[text], images=image_inputs, videos=video_inputs,
                   padding=True, return_tensors="pt", **video_kwargs).to(model.device)

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
pred = processor.tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(pred)

Tips:

VRAM: at nframes=64 / 98k–197k px, inference ≈ 6–8 GB (bf16, sdpa); measured peak ~6.8 GB at 264 frames on an 8 GB cap — 200 frames fits comfortably on 8 GB (see experiment report).
Full procedures: long videos auto-cap nframes to probe_total_frames (no OOM on short clips).
LoRA adapter (200-frame 4-bit variant) also available as shahedm2001/qwen3vl-sft-lora-* (private autosaves); merge with src/merge_lora.py if you need the adapter alone.

Intended Use & Limitations

Intended for cataract-surgery video understanding research / education. Not a medical device; do not use for direct patient care.
Trained only on the listed cataract corpus; performance outside that distribution is untested. Full-video reasoning relies on sampled frames — extremely long tails may be truncated by max_seq_length.

License

Apache 2.0 — see LICENSE in the training repo.

Citation

bibtex
@misc{qwen3vl2b-cataract-sft-2026,
  title  = {Qwen3-VL-2B Cataract Surgery SFT},
  author = {Momenzadeh, Shahed},
  howpublished = {Hugging Face},
  year   = {2026},
  note   = {Merged full-precision SFT checkpoint, QLoRA r16 on dataset_sft}
}

Base

Qwen/Qwen3-VL-2B-Instruct (qwen3_vl, patch 16, 2B)

Dataset

shahedm2001/dataset_sft — cataract surgery video, LLaVA format (video + conversations)

Train / Val

5,597 / 689 samples (2,174 train videos: 2,066 clips + 108 full procedures)

Task

Describe scene + chain-of-thought MCQ (step / visual observation / instrument), teacher/critic descriptions, timestamped full-video narration

Quantization

QLoRA 4-bit during training (NF4, double_quant, bnb_4bit_compute_dtype=bf16); merged output is full-precision bf16

Group

Argument

Value

Model

model_id

Qwen/Qwen3-VL-2B-Instruct

bits / quant_type

4 / nf4 (double_quant True)

LoRA

lora_rank / lora_alpha /

bash

.venv/bin/python -u src/train/train_sft.py --model_id Qwen/Qwen3-VL-2B-Instruct \
  --data_path data/sft_train_dataset_sft.json --eval_path data/sft_val_dataset_sft.json \
  --output_dir output/sft_lora --bits 4 --lora_enable True --vision_lora True --use_dora False \
  --lora_rank 16 --lora_alpha 32 --lora_dropout 0.05 --num_lora_modules -1 \
  --lora_namespan_exclude "['lm_head','embed_tokens','merger','pos_embed']" \
  --freeze_vision_tower True --freeze_llm True --freeze_merger False \
  --bf16 True --fp16 False --tf32 True --disable_flash_attn2 True --use_liger_kernel True \
  --num_train_epochs 2 --per_device_train_batch_size 4 --gradient_accumulation_steps 4 \
  --learning_rate 1e-4 --vision_lr 2e-6 --merger_lr 1e-5 --weight_decay 0.1 --warmup_steps 10 \
  --lr_scheduler_type cosine --video_min_pixels 98304 --video_max_pixels 196608 --nframes 64 \
  --max_seq_length 32768 --gradient_checkpointing True --image_folder dataset_sft

Metric

First

Last

Best

loss

2.166

1.062

0.6804

grad_norm

2.50 (→ peaks ~33 at step 700, cosine decay tail)

learning_rate

5.2e-10 at step 700 (cosine schedule, 2 epochs)

bash

git clone https://github.com/shahedmomenzadeh/qwen3-VL-2B-finetune.git
cd qwen3-VL-2B-finetune
bash setup.sh  # uv + .venv (python 3.12)
source .venv/bin/activate
export HF_HOME=$PWD/hf_cache PYTHONPATH=src

Package

Version

torch

2.14.0+cu130

transformers

5.17.0.dev0

peft

0.19.1

trl

1.12.0

accelerate

1.14.0

bitsandbytes

0.49.2

qwen-vl-utils

0.0.14

liger-kernel

0.8.0

python

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText
from qwen_vl_utils import process_vision_info

model_id = "shahedm2001/qwen3-vl-2b-cataract-sft"  # <-- update after publish
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    model_id, dtype=torch.bfloat16, device_map="auto",
    attn_implementation="sdpa", trust_remote_code=True)
model.eval()

video = "path/to/clip.mp4"
question = "Describe what is happening in this cataract surgical video clip."

# --- choose ONE of nframes / fps, plus desired pixel bounds ---
messages = [{"role": "user", "content": [
    {"type": "video", "video": video,
     "nframes": 64,                 # ← max frames; use one of nframes/fps
     # "fps": 1.0,                  # ← alternative: frames per second
     "min_pixels": 98_304,          # 96*32*32 — lower bound
     "max_pixels": 196_608},        # 192*32*32 — upper bound (increase for more detail / more tokens)
    {"type": "text", "text": question}
]}]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs, video_kwargs = process_vision_info(
    messages, return_video_kwargs=True)
video_kwargs = {k: v for k, v in (video_kwargs or {}).items() if k != "fps"}
inputs = processor(text=[text], images=image_inputs, videos=video_inputs,
                   padding=True, return_tensors="pt", **video_kwargs).to(model.device)

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
pred = processor.tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(pred)

bibtex

@misc{qwen3vl2b-cataract-sft-2026,
  title  = {Qwen3-VL-2B Cataract Surgery SFT},
  author = {Momenzadeh, Shahed},
  howpublished = {Hugging Face},
  year   = {2026},
  note   = {Merged full-precision SFT checkpoint, QLoRA r16 on dataset_sft}
}

qwen3-vl-2b-cataract-sft

README

Model & Data

Training Configuration

Training Curves

Environment & Reproducibility

Inference — Control `fps` / `nframes` and Resolution

Intended Use & Limitations

License

Citation

Explore FriendliAI today

README

Model & Data

Training Configuration

Training Curves

Environment & Reproducibility

Inference — Control `fps` / `nframes` and Resolution

Intended Use & Limitations

License

Citation

qwen3-vl-2b-cataract-sft

README

Model & Data

Training Configuration

Training Curves

Environment & Reproducibility

Inference — Control fps / nframes and Resolution

Intended Use & Limitations

License

Citation

Explore FriendliAI today

README

Model & Data

Training Configuration

Training Curves

Environment & Reproducibility

Inference — Control fps / nframes and Resolution

Intended Use & Limitations

License

Citation

Inference — Control `fps` / `nframes` and Resolution

Inference — Control `fps` / `nframes` and Resolution