GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
shahedm2001
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
shahedm2001
Model Tree
Qwen/Qwen3-VL-2B-Instruct
Input Modalities
Output Modalities
Supported Functionality
| Base | Qwen/Qwen3-VL-2B-Instruct (qwen3_vl, patch 16, 2B) |
| Dataset | shahedm2001/dataset_sft — cataract surgery video, LLaVA format (video + conversations) |
| Train / Val | 5,597 / 689 samples (2,174 train videos: 2,066 clips + 108 full procedures) |
| Task | Describe scene + chain-of-thought MCQ (step / visual observation / instrument), teacher/critic descriptions, timestamped full-video narration |
| Quantization | QLoRA 4-bit during training (NF4, double_quant, bnb_4bit_compute_dtype=bf16); merged output is full-precision bf16 |
| Group | Argument | Value |
|---|---|---|
| Model | model_id | Qwen/Qwen3-VL-2B-Instruct |
bits / quant_type | 4 / nf4 (double_quant True) | |
| LoRA | lora_rank / lora_alpha / |
bash
.venv/bin/python -u src/train/train_sft.py --model_id Qwen/Qwen3-VL-2B-Instruct \ --data_path data/sft_train_dataset_sft.json --eval_path data/sft_val_dataset_sft.json \ --output_dir output/sft_lora --bits 4 --lora_enable True --vision_lora True --use_dora False \ --lora_rank 16 --lora_alpha 32 --lora_dropout 0.05 --num_lora_modules -1 \ --lora_namespan_exclude "['lm_head','embed_tokens','merger','pos_embed']" \ --freeze_vision_tower True --freeze_llm True --freeze_merger False \ --bf16 True --fp16 False --tf32 True --disable_flash_attn2 True --use_liger_kernel True \ --num_train_epochs 2 --per_device_train_batch_size 4 --gradient_accumulation_steps 4 \ --learning_rate 1e-4 --vision_lr 2e-6 --merger_lr 1e-5 --weight_decay 0.1 --warmup_steps 10 \ --lr_scheduler_type cosine --video_min_pixels 98304 --video_max_pixels 196608 --nframes 64 \ --max_seq_length 32768 --gradient_checkpointing True --image_folder dataset_sft
| Metric | First | Last | Best |
|---|---|---|---|
| loss | 2.166 | 1.062 | 0.6804 |
| grad_norm | 2.50 (→ peaks ~33 at step 700, cosine decay tail) | ||
| learning_rate | 5.2e-10 at step 700 (cosine schedule, 2 epochs) |
700 log lines (output/logs/sft/losses.csv); wall time ≈ 11h on 1× 49 GB GPU (49140 MiB, sdpa attention, bf16).
bash
git clone https://github.com/shahedmomenzadeh/qwen3-VL-2B-finetune.gitcd qwen3-VL-2B-finetunebash setup.sh # uv + .venv (python 3.12)source .venv/bin/activateexport HF_HOME=$PWD/hf_cache PYTHONPATH=src| Package | Version |
|---|---|
| torch | 2.14.0+cu130 |
| transformers | 5.17.0.dev0 |
| peft | 0.19.1 |
| trl | 1.12.0 |
| accelerate | 1.14.0 |
| bitsandbytes | 0.49.2 |
| qwen-vl-utils | 0.0.14 |
| liger-kernel | 0.8.0 |
fps / nframes and ResolutionThis is a standard transformers + qwen_vl_utils checkpoint. No quantization needed.
python
import torchfrom transformers import AutoProcessor, AutoModelForImageTextToTextfrom qwen_vl_utils import process_vision_info model_id = "shahedm2001/qwen3-vl-2b-cataract-sft" # <-- update after publishprocessor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)model = AutoModelForImageTextToText.from_pretrained( model_id, dtype=torch.bfloat16, device_map="auto", attn_implementation="sdpa", trust_remote_code=True)model.eval() video = "path/to/clip.mp4"question = "Describe what is happening in this cataract surgical video clip." # --- choose ONE of nframes / fps, plus desired pixel bounds ---messages = [{"role": "user", "content": [ {"type": "video", "video": video, "nframes": 64, # ← max frames; use one of nframes/fps # "fps": 1.0, # ← alternative: frames per second "min_pixels": 98_304, # 96*32*32 — lower bound "max_pixels": 196_608}, # 192*32*32 — upper bound (increase for more detail / more tokens) {"type": "text", "text": question}]}] text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)image_inputs, video_inputs, video_kwargs = process_vision_info( messages, return_video_kwargs=True)video_kwargs = {k: v for k, v in (video_kwargs or {}).items() if k != "fps"}inputs = processor(text=[text], images=image_inputs, videos=video_inputs, padding=True, return_tensors="pt", **video_kwargs).to(model.device) with torch.no_grad(): out = model.generate(**inputs, max_new_tokens=256, do_sample=False)pred = processor.tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)print(pred)Tips:
nframes=64 / 98k–197k px, inference ≈ 6–8 GB (bf16, sdpa); measured peak ~6.8 GB at 264 frames on an 8 GB cap — 200 frames fits comfortably on 8 GB (see experiment report).nframes to probe_total_frames (no OOM on short clips).shahedm2001/qwen3vl-sft-lora-* (private autosaves); merge with src/merge_lora.py if you need the adapter alone.max_seq_length.Apache 2.0 — see LICENSE in the training repo.
bibtex
@misc{qwen3vl2b-cataract-sft-2026, title = {Qwen3-VL-2B Cataract Surgery SFT}, author = {Momenzadeh, Shahed}, howpublished = {Hugging Face}, year = {2026}, note = {Merged full-precision SFT checkpoint, QLoRA r16 on dataset_sft}}.venv/bin/python -u src/train/train_sft.py --model_id Qwen/Qwen3-VL-2B-Instruct \ --data_path data/sft_train_dataset_sft.json --eval_path data/sft_val_dataset_sft.json \ --output_dir output/sft_lora --bits 4 --lora_enable True --vision_lora True --use_dora False \ --lora_rank 16 --lora_alpha 32 --lora_dropout 0.05 --num_lora_modules -1 \ --lora_namespan_exclude "['lm_head','embed_tokens','merger','pos_embed']" \ --freeze_vision_tower True --freeze_llm True --freeze_merger False \ --bf16 True --fp16 False --tf32 True --disable_flash_attn2 True --use_liger_kernel True \ --num_train_epochs 2 --per_device_train_batch_size 4 --gradient_accumulation_steps 4 \ --learning_rate 1e-4 --vision_lr 2e-6 --merger_lr 1e-5 --weight_decay 0.1 --warmup_steps 10 \ --lr_scheduler_type cosine --video_min_pixels 98304 --video_max_pixels 196608 --nframes 64 \ --max_seq_length 32768 --gradient_checkpointing True --image_folder dataset_sftgit clone https://github.com/shahedmomenzadeh/qwen3-VL-2B-finetune.gitcd qwen3-VL-2B-finetunebash setup.sh # uv + .venv (python 3.12)source .venv/bin/activateexport HF_HOME=$PWD/hf_cache PYTHONPATH=src@misc{qwen3vl2b-cataract-sft-2026, title = {Qwen3-VL-2B Cataract Surgery SFT}, author = {Momenzadeh, Shahed}, howpublished = {Hugging Face}, year = {2026}, note = {Merged full-precision SFT checkpoint, QLoRA r16 on dataset_sft}}lora_dropout| 16 / 32 / 0.05 |
target_modules | LLM q/k/v/o + gate/up/down + vision linears (num_lora_modules -1, exclude lm_head, embed_tokens, merger, pos_embed — merger full-trainable) |
freeze_vision_tower / freeze_llm / freeze_merger | True / True / False |
| Opt | learning_rate / vision_lr / merger_lr | 1e-4 / 2e-6 / 1e-5 |
weight_decay / warmup_steps / lr_scheduler | 0.1 / 10 / cosine |
bf16 / tf32 / use_liger_kernel / gradient_checkpointing | True / True / True / True |
| Batch | per_device_train_batch_size / gradient_accumulation_steps | 4 / 4 → global 16 |
num_train_epochs | 2 (700 steps), save_steps 100, save_total_limit 3 |
| Video | nframes | 64 (even-capped to probe_total_frames, min 2) |
video_min_pixels / video_max_pixels | 98,304 / 196,608 |
max_seq_length | 32,768 |
dataloader_num_workers / prefetch | 2 / 2 |