yah01

vjev-vision

README

License: apache-2.0

Use it

The web console and the HTTP API live in BubbleCal/vjev-serve:

bash

pip install git+https://github.com/BubbleCal/vjev-servevjev-serve --model yah01/vjev-vision        # then open http://localhost:8800

Or, with nothing but transformers, the single file in this repo:

python

from huggingface_hub import hf_hub_downloadimport importlib.utilspec = importlib.util.spec_from_file_location("vjev_infer", hf_hub_download("yah01/vjev-vision", "vjev_infer.py"))vjev_infer = importlib.util.module_from_spec(spec); spec.loader.exec_module(vjev_infer) m = vjev_infer.Vjev("yah01/vjev-vision")          # cuda / mps / cpum.ask(state=["photo.jpg", "Frame from the warehouse camera, 12:40."],      questions={"person": {"type": "noul",   "instructions": "There is a person in this image."},                 "where":  {"type": "choice", "instructions": "Where is the forklift?",                            "criteria": {"left": "left half", "right": "right half", "none": "no forklift"}},                 "busy":   {"type": "score",  "instructions": "How cluttered is the scene?",                            "criteria": ["empty", "sparse", "busy", "crowded"]}})

Options inside one question compete (one softmax); questions never see each other. To rate several independent things, ask one noul per thing (vjev-serve's multilabel type does this). About 9 GB of memory in bf16/fp16.

What is in this repo

Table with columns: file, what
file	what
`model-*.safetensors`, `config.json`, tokenizer and processor files	the full model, bf16: Qwen3.5-4B with the checkpoint's LoRA merged in
`head.pt`	the listwise scoring head: one shared linear layer read at each option's slot
`vjev.json`	how inputs are rendered (`readout: trailing`, `pause: 0`, length budgets)
`vjev_infer.py`	stand-alone inference
`adapter/`	the same weights as a LoRA adapter (PEFT, r=32, α=64) over `Qwen/Qwen3.5-4B`, fitted on the nf4-quantized base

The merged weights are the nf4-dequantized base the adapter was trained against, plus the adapter, in bf16 — no bitsandbytes needed. Against the adapter on its nf4 base (CUDA, 118 questions over 10 images): max |Δp| 0.0097, mean 0.0036, no answer changed.

How it was trained

Text (3,868 steps): QLoRA on ~145k typed questions with soft labels from the Jev API plus human-labelled corpora; KL to the teacher distribution for choice/score, soft BCE for noul. All options of a question share one sequence and are read at a trailing Answer: (A) (B) … slot, so options can see each other.
Vision (this checkpoint, step 600 of a 1,200-step run warm-started from the text stage): a second LoRA on the vision tower (0.1× the text learning rate), on ~84k geometry questions from COCO-2017 annotations (which object is highest / smallest / left of…, counts, presence with adversarial absent objects; soft labels by Monte-Carlo perturbation of the boxes) and ~15k VQAv2 questions with their 10-annotator answer distributions, mixed with 25% text.

Results (held out: images and texts never seen in training)

Table with columns: text stage (zero-shot), pilot, 300 steps, this, step 600, final step 1200
	text stage (zero-shot)	pilot, 300 steps	this, step 600	final step 1200
COCO geometry, choice accuracy	0.506	0.665	0.735	0.703
COCO geometry, choice ECE ↓	0.171	0.094	0.183	0.148
VQAv2, choice accuracy	0.579	0.654	0.706	0.721

Step 600 was picked over the final step: the final step called absent objects present twice as often, for a small VQA gain. On text the vision stage cost nothing; the teacher itself scores 0.797 accuracy and 0.102 ECE on the same rows.

Options interact as in the teacher: adding a competing option protects the leader and takes its mass from the runners-up (a paired effect of +0.0066 over the order-permutation noise floor, CI excluding 0) — something a one-option-per-pass scorer cannot do.

Limits

Spatial answers are over-confident (ECE 0.18 on COCO geometry): the ranking is better than the probabilities.
Trained on single images; several images in one request (Picture 1:, Picture 2:) rely on the base model's ability.
Options are read in the order given; reordering them moves probabilities by a TVD of ~0.037 on average.
Images are resized to a 512 px longer side, the training resolution.

License

Apache-2.0. The base model, Qwen3.5-4B, is Apache-2.0.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

yah01

Model Tree

Base

Qwen/Qwen3.5-4B

Fine-tuned

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality