chenyeikai

VisProbe-Qwen3-VL-2B-LoRA

README

License: apache-2.0

Results — frozen baseline vs this adapter

Same test sets, same scoring protocol, single RTX 3090, bf16. Accuracy (and absolute gain):

Table with columns: Test set, Baseline, After LoRA, Gain
Test set	Baseline	After LoRA	Gain
SNLI-VE (visual entailment, 3-way, n=21,482)	0.6918	0.8998	+20.80 pp
VQAv2 yes/no (seeded 1/3 subset, n=26,846)	0.6618	0.8558	+19.40 pp
Visual7W (4-way multiple choice, n=6,695)	0.8512	0.8875	+3.63 pp
Macro average (equal weight)	0.7350	0.8810	+14.6 pp

Full accuracy/ECE/Brier tables and loss curves: see results/SUMMARY.md and EXPERIMENTS.md in the VisProbe repository. This is the largest gain of all released VisProbe adapters — the best accuracy-per-compute point.

Training

Table with columns: Item, Value
Item	Value
Data	99,999 mixed examples (33,333 per dataset, stratified by label, seed=42)
Epochs / steps	1 / 25,000 (batch size 4; per-example micro-batches with gradient accumulation)
LoRA	r=8, alpha=16, dropout=0.05, bias=none
Target modules	`q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj` (language decoder only; vision tower frozen)
Optimisation	lr 1e-4, 3% linear warmup + cosine decay, grad clip 1.0, AdamW
Precision	bfloat16, gradient checkpointing (`use_reentrant=False`)
Checkpoint

Trainable parameters: ~18.6M (adapter weights only, 34.9 MB in safetensors).

How to use

With VisProbe (recommended — gives probabilities, not text)

python

from vlm.scoring.vlm_scorer import VLMScorer # The adapter id is downloaded from the Hub automatically; the base model is Qwen3-VL-2B-Instructscorer = VLMScorer("qwen3vl2b", adapter="chenyeikai/VisProbe-Qwen3-VL-2B-LoRA")result = scorer.score(image, fields, context_text=context)# result["fields"][name]["scores"] -> {"A": p, "B": p, ...} candidate probabilities

Generic PEFT loading

python

import torchfrom peft import PeftModelfrom transformers import AutoModelForImageTextToText, AutoProcessor base = AutoModelForImageTextToText.from_pretrained(    "Qwen/Qwen3-VL-2B-Instruct", torch_dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(base, "chenyeikai/VisProbe-Qwen3-VL-2B-LoRA")processor = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-2B-Instruct")

Scope & limitations

The adapter is specialised for closed-set choice / boolean questions rendered with the VisProbe prompt schema (single-token answer codes). It is not a general-purpose chat upgrade.
Training data are three academic visual-reasoning benchmarks; in-domain behaviour is measured by the tables above, out-of-domain behaviour is not.
Post-finetune calibration error (ECE) changes slightly (still small). VisProbe ships a temperature-calibration utility, but calibrated-temperature results are not part of this release.
VQAv2 results use a seeded random 1/3 test subset (standard error ≈ ±0.5 pp).

License

Adapter weights: Apache-2.0, following the base model Qwen3-VL-2B-Instruct. Datasets remain under their upstream licenses (Flickr30k / COCO / Visual Genome based).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

chenyeikai

Model Tree

Base

Qwen/Qwen3-VL-2B-Instruct

Adapter

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Results — frozen baseline vs this adapter

Same test sets, same scoring protocol, single RTX 3090, bf16. Accuracy (and absolute gain):

Table with columns: Test set, Baseline, After LoRA, Gain
Test set	Baseline	After LoRA	Gain
SNLI-VE (visual entailment, 3-way, n=21,482)	0.6918	0.8998	+20.80 pp
VQAv2 yes/no (seeded 1/3 subset, n=26,846)	0.6618	0.8558	+19.40 pp
Visual7W (4-way multiple choice, n=6,695)	0.8512	0.8875	+3.63 pp
Macro average (equal weight)	0.7350	0.8810	+14.6 pp

Training

Table with columns: Item, Value
Item	Value
Data	99,999 mixed examples (33,333 per dataset, stratified by label, seed=42)
Epochs / steps	1 / 25,000 (batch size 4; per-example micro-batches with gradient accumulation)
LoRA	r=8, alpha=16, dropout=0.05, bias=none
Target modules	`q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj` (language decoder only; vision tower frozen)
Optimisation	lr 1e-4, 3% linear warmup + cosine decay, grad clip 1.0, AdamW
Precision	bfloat16, gradient checkpointing (`use_reentrant=False`)
Checkpoint

Trainable parameters: ~18.6M (adapter weights only, 34.9 MB in safetensors).

How to use

With VisProbe (recommended — gives probabilities, not text)

python

from vlm.scoring.vlm_scorer import VLMScorer # The adapter id is downloaded from the Hub automatically; the base model is Qwen3-VL-2B-Instructscorer = VLMScorer("qwen3vl2b", adapter="chenyeikai/VisProbe-Qwen3-VL-2B-LoRA")result = scorer.score(image, fields, context_text=context)# result["fields"][name]["scores"] -> {"A": p, "B": p, ...} candidate probabilities

Generic PEFT loading

python

import torchfrom peft import PeftModelfrom transformers import AutoModelForImageTextToText, AutoProcessor base = AutoModelForImageTextToText.from_pretrained(    "Qwen/Qwen3-VL-2B-Instruct", torch_dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(base, "chenyeikai/VisProbe-Qwen3-VL-2B-LoRA")processor = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-2B-Instruct")

Scope & limitations

The adapter is specialised for closed-set choice / boolean questions rendered with the VisProbe prompt schema (single-token answer codes). It is not a general-purpose chat upgrade.
Training data are three academic visual-reasoning benchmarks; in-domain behaviour is measured by the tables above, out-of-domain behaviour is not.
Post-finetune calibration error (ECE) changes slightly (still small). VisProbe ships a temperature-calibration utility, but calibrated-temperature results are not part of this release.
VQAv2 results use a seeded random 1/3 test subset (standard error ≈ ±0.5 pp).

License

Adapter weights: Apache-2.0, following the base model Qwen3-VL-2B-Instruct. Datasets remain under their upstream licenses (Flickr30k / COCO / Visual Genome based).