GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
shravvvv
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
shravvvv
Model Tree
Qwen/Qwen3-VL-2B-Instruct
Input Modalities
Output Modalities
Supported Functionality
This is a LoRA adapter, so load the base model first and attach the adapter:
python
import torchfrom PIL import Imagefrom transformers import AutoModelForVision2Seq, AutoProcessorfrom peft import PeftModel BASE = "Qwen/Qwen3-VL-2B-Instruct"ADAPTER = "shravvvv/VISE" model = AutoModelForVision2Seq.from_pretrained(BASE, torch_dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(model, ADAPTER)processor = AutoProcessor.from_pretrained(ADAPTER)model.eval() image = Image.open("example.jpg").convert("RGB")messages = [{"role": "user", "content": [ {"type": "image", "image": image}, {"type": "text", "text": "Describe this image in detail."},]}]text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device) with torch.inference_mode(): out = model.generate(**inputs, max_new_tokens=128)print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])| Metric | Base | VISE |
|---|---|---|
| COCO (CIDEr) | 21.54 | 38.39 |
| NoCaps (CIDEr) | 19.52 | 34.25 |
| Flickr30k (CIDEr) | 26.09 | 42.64 |
| TextCaps (CIDEr) | 22.20 | 41.86 |
| CHAIR-I (lower is better) | 13.21 | 8.21 |
| CHAIR-S (lower is better) | 45.96 | 40.51 |
| POPE Accuracy | 89.01 | 90.03 |
| ScienceQA | 79.42 | 83.61 |
VISE improves captioning, VQA, reasoning, and hallucination together with no task tradeoffs, and the same recipe generalizes across larger scales and other backbone families. Full results are in our paper.
Qwen/Qwen3-VL-2B-Instruct, vision encoder frozen.r=16, alpha=32, dropout=0.05 on the attention, MLP, and projector layers.lr=1e-6, weight decay 0.01, gradient clipping 1.0, bfloat16.0.020), reward weights 0.5 / 0.5.Apache 2.0.
bibtex
@inproceedings{venkatraman2026vise, title = {Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models}, author = {Venkatraman, Shravan and Thawkar, Ritesh and Thawakar, Omkar and Anwer, Rao Muhammad and Cholakkal, Hisham and Khan, Salman and Khan, Fahad Shahbaz}, booktitle = {European Conference on Computer Vision (ECCV)}, year = {2026}}import torchfrom PIL import Imagefrom transformers import AutoModelForVision2Seq, AutoProcessorfrom peft import PeftModel BASE = "Qwen/Qwen3-VL-2B-Instruct"ADAPTER = "shravvvv/VISE" model = AutoModelForVision2Seq.from_pretrained(BASE, torch_dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(model, ADAPTER)processor = AutoProcessor.from_pretrained(ADAPTER)model.eval() image = Image.open("example.jpg").convert("RGB")messages = [{"role": "user", "content": [ {"type": "image", "image": image}, {"type": "text", "text": "Describe this image in detail."},]}]text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device) with torch.inference_mode(): out = model.generate(**inputs, max_new_tokens=128)print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])@inproceedings{venkatraman2026vise, title = {Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models}, author = {Venkatraman, Shravan and Thawkar, Ritesh and Thawakar, Omkar and Anwer, Rao Muhammad and Cholakkal, Hisham and Khan, Salman and Khan, Fahad Shahbaz}, booktitle = {European Conference on Computer Vision (ECCV)}, year = {2026}}