Measured decision quality
89.3% macro accuracy · 0.896 AUROC · 0.285 calibrated NLL

Four benchmarks, equal-weight macro averages, November 2026. Visual Jev uses task SFT; API training data are unknown. TypeSafe Jev leads on ECE and TextVQA. NLL follows task-local temperature scaling. Results and protocol ↗
Test images only (70% of each benchmark's images, split by a fixed hash of the image id); the same questions and option order for every system; each system gets its own temperature fitted on the same calibration images. TallyQA is the full test build with GQA-training duplicates removed. API probabilities are returned to two decimals and floored at 0.005; a failed request counts as a uniform guess. NLL and ECE after temperature scaling; AUROC of the top probability for correct vs. wrong answers. Macro is the equal-weight mean of the four benchmarks.
Table with columns: System, GQA, SNLI-VE, TextVQA, TallyQA, Macro, NLL (TS), ECE (TS), AUROC| System | GQA | SNLI-VE | TextVQA | TallyQA | Macro | NLL (TS) | ECE (TS) | AUROC |
|---|
| Visual Jev 8B | 0.919 | 0.818 | 0.979 | 0.855 | 0.893 | 0.285 | 0.021 | 0.896 |
| Visual Jev 4B | 0.916 | 0.808 | 0.975 | 0.841 | 0.885 | 0.309 | 0.025 | 0.890 |
| TypeSafe Jev | 0.885 | 0.713 | 0.982 | 0.851 | 0.858 | 0.361 | 0.013 | 0.870 |
| OpenAI Decisions | 0.834 | 0.732 | 0.955 | 0.818 | 0.835 | 0.440 | 0.020 | 0.833 |
We sent the same test questions, with the options in the same order, to two
hosted APIs that also return one probability per option: OpenAI's
GPT-6 Luna Decisions (openai/gpt-6-luna-decisions-20261006, through
OpenRouter) and TypeSafe Jev (jev-1.14.0, whose image input TypeSafe
marks as not yet production quality), queried in November 2026. Every system
gets its own temperature, fitted on the same calibration images.
Read it with three caveats:
- The adapters were trained on GQA and SNLI-VE training data; API training data are unknown. Most of the adapters' lead comes from those two tasks. On the
two task families no post-training saw (
†), the systems are close, and
TypeSafe Jev has the best TextVQA accuracy.
- Both APIs return probabilities rounded to two decimals; we floor them at
0.005 before taking logs. A request that failed (1.9% of questions for
TypeSafe Jev, which rejected some image sizes; 0.3% for GPT-6 Luna,
all refusals) counts as a uniform guess rather than being dropped.
- TypeSafe Jev is not deterministic: sending the identical request again
changed its top answer for 0.8% of GQA, 2.7% of SNLI-VE and 10.3% of
short-TallyQA questions. GPT-6 Luna returned identical probabilities on
every repeat.
Source: checked-in API comparison data.
Try the released model
CUDA or Apple Silicon MPS · Inference only · No training dataset required
git clone https://github.com/guanxuyu-sv/Visual-Jev.gitcd Visual-Jevpython3 -m venv .venv && source .venv/bin/activatepip install torch torchvisionpip install -r code/requirements.txtpython code/examples/quickstart.py \ --model Qwen/Qwen3-VL-8B-Instruct \ --adapter guanxuyu/visual-jev-8b-answer-sft \ --image assets/figures/demo_bottle.jpg \ --request-file code/examples/bottle_questions.json --device auto
The runner downloads the backbone and this adapter on first use. --device auto selects CUDA first, then Apple MPS. For CUDA, install a compatible PyTorch build before the remaining dependencies; on Apple Silicon, use the regular macOS build shown above.
The request supplies named questions, shared state, and 2–16 candidate choices per question. Evaluation covers at most eight choices. The runner caches the shared input and returns a choice and probabilities for each question.
Explore the project’s interactive 4B reference examples ↗
Macro over the same four benchmarks:
Table with columns: Original 8B, This adapter | Original 8B | This adapter |
|---|
| NLL, raw | 0.634 | 0.298 |
| NLL, after temperature scaling | 0.376 | 0.285 |
| ECE (15 bins), after temperature scaling | 0.022 | 0.021 |
| AUROC, correct vs. wrong by top probability | 0.851 | 0.896 |
Post-training mainly improves ranking: confident answers are more often
right (AUROC, which does not depend on accuracy). It does not make the raw
probabilities calibrated; the fitted temperatures are all above 1, i.e. the
model is overconfident. A single temperature per task fixes that.
temperatures.json lists the temperature fitted for each seed and benchmark
on a separate calibration split; apply it as softmax(logits / T) over the
supplied options. For a new task, fit your own temperature on a few hundred
labelled questions from that task: in our measurements on the 4B adapter,
temperatures transferred from other tasks calibrated clearly worse.
Two further cautions before acting on the probabilities with a threshold:
- Option order matters. On the 4B adapter, listing the same options in a
different order changed some thresholded decisions, at a rate well above that
of changing the inference backend; we did not repeat that study at 8B. Fix the
option order you deploy with, and audit it.
- Probabilities are relative to the options you supply; the model has no
built-in way to say that the right answer is missing. Add an option such as
incorrect_question or not enough evidence if you need one.
Research details
Test images only (70% of each benchmark's images, split by a fixed hash of the
image id); mean ± standard deviation over three seeds. † marks a task family
no post-training saw.
Table with columns: Benchmark, Original 8B, This adapter, 4B adapter, n test| Benchmark | Original 8B | This adapter | 4B adapter | n test |
|---|
| GQA-Choice | 0.885 | 0.919 ± 0.001 | 0.916 | 5,447 |
| SNLI-VE | 0.695 | 0.818 ± 0.004 | 0.808 | 1,368 |
TextVQA-Choice † | 0.979 | 0.979 ± 0.003 | 0.975 | 1,742 |
TallyQA-Choice † | 0.853 | 0.855 ± 0.010 | 0.841 | 3,681 |
| Macro | 0.853 | 0.893 ± 0.003 | 0.885 | |
The gain sits on the two task families represented in training; on the two
held-out families the adapter matches the untrained backbone. The TallyQA
column uses the full TallyQA test build (natural count distribution, with the
images that are copies of GQA training images removed), distinct from the short count-balanced TallyQA subset in the original ablation evaluation. Compare this table with the full-build API comparison on the 4B card, not its original ablation table.
import torchfrom transformers import AutoModelForImageTextToText, AutoProcessorfrom peft import PeftModel BASE = "Qwen/Qwen3-VL-8B-Instruct"ADAPTER = "guanxuyu/visual-jev-8b-answer-sft" # seed 0 lives at the root model = AutoModelForImageTextToText.from_pretrained(BASE, dtype=torch.bfloat16, device_map="cuda")model = PeftModel.from_pretrained(model, ADAPTER).eval()processor = AutoProcessor.from_pretrained(BASE) # the other two seeds:# PeftModel.from_pretrained(model, ADAPTER, subfolder="seed1")
The scores are candidate-token logits read from the backbone's own LM head at
one fixed readout position, not free-form generation. Prompt layout, readout
position and candidate tokenisation determine the result; use the prompt
construction in the code repository
to reproduce the numbers above.
Table | |
|---|
| Base | Qwen/Qwen3-VL-8B-Instruct, vision tower frozen (exclude_modules: ".*visual.*") |
| Method | LoRA on the language tower, r=16, α=32, dropout 0.05 |
| Targets | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Trainable | 43,646,976 parameters |
| Objective | full-vocabulary next-token cross-entropy at the readout position |
| Steps | 3,000, batch size 8, grad checkpointing, lr 1e-4, 100 warmup, bf16 |
| Images | native resolution (see note) |
| Data | question records derived from GQA and SNLI-VE training splits; GQA Choice items presented with K in {2, 4} options |
| Seeds | 0 (root), 1 (seed1/), 2 (seed2/) |
Note on image resolution. The training configuration sets
max_pixels = 200,704 (196 visual tokens), but the Transformers image
processor used ignores that attribute, so training and every evaluation above
ran at the processor's default, which keeps native resolution (a 500×500 GQA
image becomes 256 visual tokens). The same correction applies to the 4B adapter.
No new human annotation was collected. The training records are derived from
publicly released benchmarks; no images are redistributed here, and the
adapter is a set of low-rank parameter deltas carrying no image data. SNLI-VE
is built on Flickr30k images, which are licensed for non-commercial research;
using the backbone requires accepting its own licence.
Limitations
- Helps only on task families seen in training (see the results table).
- Calibration needs labelled data from the target task; thresholds fitted on
it are empirical, not guarantees.
- Decisions depend on the option order.
- Single-machine study on one consumer GPU; English only; at most 8 options
evaluated.
Citation
@misc{yu2026visualjev, title = {Visual Jev: Accurate and Efficient Decisions from Shared Visual Context}, author = {Guanxu Yu and Yuhang Yao}, year = {2026}, eprint = {2609.25845}, archivePrefix = {arXiv}, primaryClass = {cs.CV}, url = {https://arxiv.org/abs/2609.25845}}
License and resources
This adapter is released under Apache-2.0. The backbone and source datasets retain their respective terms.
Reproduction guide · Experimental outputs · Brand asset sources