guanxuyu

visual-jev-4b-answer-sft

README

License: apache-2.0

Try it

The simplest way to try the adapter is with the repository's inference quickstart. It downloads the Qwen3-VL-4B base and this adapter from Hugging Face on first run. The included example asks six different questions about one image and includes an incorrect_question choice for false assumptions (selected with 0.966 probability for the dog/person mismatch shown above).

bash

git clone https://github.com/guanxuyu-sv/Visual-Jev.gitcd Visual-Jevpython3 -m venv .venv && source .venv/bin/activatepip install torch torchvisionpip install -r code/requirements.txtpython code/examples/quickstart.py \    --image assets/figures/demo_bottle.jpg \    --request-file code/examples/bottle_questions.json --device auto

On Apple Silicon, --device auto selects MPS. For CUDA, install a PyTorch build that matches your CUDA runtime before installing the remaining requirements. See the reproduction guide for the training and evaluation workflow.

This is the system our paper recommends, and it is deliberately the simplest one we trained. We also built typed decision heads on top of the same interface and measured no accuracy advantage from them (+0.000 macro against this adapter at matched data and budget, three seeds each). Those heads are therefore not part of this release. The finding is the reason this adapter is what we publish: adapt the backbone, keep the readout the backbone already has.

What it improves, and what it does not

Equal-weight macro accuracy over four benchmarks, mean ± spread over three seeds. † marks a benchmark no post-training ever saw.

Table with columns: Benchmark, Original backbone, This adapter, n
Benchmark	Original backbone	This adapter	n
GQA-Choice	0.880	0.916 ± 0.002	5,447
SNLI-VE	0.629	0.808 ± 0.006	1,368
TextVQA-Choice `†`	0.974	0.975 ± 0.002	1,742
TallyQA-Choice `†`	0.340

Read the table honestly: the entire gain sits on the two task families represented in training. On the two held-out families the adapter is indistinguishable from the untrained backbone — TextVQA moves by 0.001 and TallyQA by 0.005 against a seed spread of 0.014. If your task resembles relational VQA or visual entailment, this adapter helps; if it resembles counting, it does not, and TallyQA at 0.345 against a six-option chance rate of 0.167 is weak in absolute terms for both models.

Loading

python

import torchfrom transformers import AutoModelForImageTextToText, AutoProcessorfrom peft import PeftModel BASE = "Qwen/Qwen3-VL-4B-Instruct"ADAPTER = "guanxuyu/visual-jev-4b-answer-sft"   # seed 0 lives at the root model = AutoModelForImageTextToText.from_pretrained(BASE, dtype=torch.bfloat16,                                                    device_map="cuda")model = PeftModel.from_pretrained(model, ADAPTER).eval()processor = AutoProcessor.from_pretrained(BASE) # the other two seeds:#   PeftModel.from_pretrained(model, ADAPTER, subfolder="seed1")

Reproducing the numbers above requires more than these weights

The scores are candidate-token logits read from the backbone's own LM head at one fixed readout position, not free-form generation. Prompt layout, readout position, and candidate tokenisation determine the result, so a custom prompt measures a different setup. The public code repository includes the prompt construction, inference path, and paper reproduction guide; use those to reproduce the reported procedure.

Two further cautions from our own measurements:

Decisions were measured in bfloat16. Path-to-path logit deviations we traced are arithmetic, not modelling: they collapse by four orders of magnitude in float32, and the worst disagreeing cases have a decision margin of 0.000.
This adapter was trained on two candidate counts only, K in {2, 4}, while the evaluation above presents K from 2 to 8. That gap matters less here than it would for a typed head: the readout is candidate tokens through the backbone's own LM head, with no fixed slot per option, so there are no slots left without gradient. Measured on the varied-K GQA evaluation it reaches 0.916, against 0.919 for a slot-indexed head trained on the varied-K data itself. Treat wider candidate sets as untested coverage rather than as a known failure.

Training

Table

Base	`Qwen/Qwen3-VL-4B-Instruct`, vision tower frozen (`exclude_modules: ".visual."`)
Method	LoRA on the language tower, r=16, α=32, dropout 0.05
Targets	`q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj`
Trainable	33,030,144 parameters
Objective	full-vocabulary next-token cross-entropy at the readout position
Steps	3,000, batch size 8, grad checkpointing, lr 1e-4, 100 warmup
Images

No new human annotation was collected. The training records are derived from publicly released benchmarks; no images are redistributed here, and the adapter is a set of low-rank parameter deltas carrying no image data. Using it requires obtaining the backbone under its own licence.

Limitations

Helps only on task families seen in training (see the table).
Counting remains weak in absolute terms.
Single-machine study on consumer GPUs; all figures are reported with seed spread and sample sizes.
English only.
Choice mode normalizes over the options supplied for each question; it has no built-in abstention outside that set. To let it reject a false premise or insufficient evidence, include an option such as incorrect_question or not enough evidence. The quickstart's incorrect_question option catches the dog/person mismatch with 0.966 probability. That score is relative to the supplied choices, not a calibrated measure of image evidence.
Separately, our trained evidence-sufficiency output, which detects when relevant image evidence has been removed, reduced macro accuracy by 0.013. That result concerns a separate output objective; it does not rule out request-time reject options.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

guanxuyu

Model Tree

Base

Qwen/Qwen3-VL-4B-Instruct

Adapter

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer