Kaylee0501

Qwen3-VL-8B-CrossImage-SFT

README

License: apache-2.0

How it was built

Supervised fine-tuning.
- Data: 36,495 chain-of-thought traces over multi-image questions (see the dataset card).
- Traces were generated by GPT teachers without access to the gold answer. A trace was kept only if the teacher's first answer was correct.
- What was trained: full fine-tuning of the language model and the vision-language merger. The vision encoder was frozen.
- Settings: 1 epoch, learning rate 1e-5 with a cosine schedule, effective batch size 16, bf16 autocast over fp32 master weights.
- Images were capped at 1280·28·28 pixels each, at both training and evaluation time.
Weight interpolation. The final weights are θ = 0.3·θ_SFT + 0.7·θ_base.
- The SFT model improved multi-image skills but weakened some base skills, for example abstaining on unanswerable items and some single-image perception. Interpolation keeps most of the gain and recovers those skills.
- The value α = 0.3 was chosen from {0.3, 0.5, 0.7, 1.0} on a development split only: a stratified 25% of the evaluation items.
- The selection rule was fixed in advance: pick the candidate that beats the base model on the most benchmarks, breaking ties by mean gain.

Results

The same harness, prompt and greedy decoding were used for every model. Scoring is letter- or exact-match, with no LLM judge.

Held-out 75% of items (never used for any choice; this is the unbiased comparison):

Table with columns: Benchmark, Qwen3-VL-8B-Instruct, this model, Δ, exact McNemar p
Benchmark	Qwen3-VL-8B-Instruct	this model	Δ	exact McNemar p
BLINK (val)	63.7	67.6	+3.9	0.001
MIRB (multi-image subsets)	54.2	59.5	+5.3	0.001
MuirBench	75.1	76.1	+1.0	0.28

Full benchmarks. These include the 25% development items, so they are slightly optimistic:

Table with columns: BLINK, MuirBench, MIRB, Mantis
	BLINK	MuirBench	MIRB	Mantis
Qwen3-VL-8B-Instruct	63.6	73.5	55.2	77.0
SFT only (α = 1.0)	63.4	72.5	61.1	79.7
this model (α = 0.3)	67.3	74.4	60.0

Evaluation sets:

BLINK val: 1,901 items
MuirBench: 2,600 items
MIRB: 814 items across the multi-image subsets 3d_scene, analogy, attribute, count, food, image_jigsaw, sightseeing and visual_chain
Mantis-Eval: 217 items

Decontamination: every training sample containing an image near-identical to any evaluation image was removed before training (dHash, then pixel check).

Prompt format (use it for best results)

System prompt:

text

You are a careful visual reasoning assistant. Reason across all the images, grounding each observation in a specific image by index, then give the final answer inside <answer></answer> tags.

User turn:

All images first.
Then the question, with options written as A. … lines.
Then this instruction:

text

Reason step by step in this exact format:Understand: state what the question asks and what form the answer must take.Perceive: one line per image, "- Image k: ...", giving only the evidence in that image which bears on the question.Connect: say how the evidence across specific images combines or conflicts, citing at least two image indices. If you pick one image, name the closest rival and why it fails.Conclude: one sentence.Then give the final answer alone inside <answer></answer> tags, with no explanation and no placeholder text inside the tags.

python

from transformers import AutoModelForImageTextToText, AutoProcessorfrom PIL import Image model_id = "Kaylee0501/Qwen3-VL-8B-CrossImage-SFT"model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="bfloat16", device_map="auto")processor = AutoProcessor.from_pretrained(model_id) SYSTEM = "You are a careful visual reasoning assistant. Reason across all the images, grounding each observation in a specific image by index, then give the final answer inside <answer></answer> tags."COT = "\n\nReason step by step in this exact format:\nUnderstand: ...\n..."   # paste the instruction above images = [Image.open("a.jpg"), Image.open("b.jpg")]question = "Do Image 1 and Image 2 show the same building?\nA. Yes\nB. No"messages = [{"role": "system", "content": [{"type": "text", "text": SYSTEM}]},            {"role": "user", "content": [{"type": "image"} for _ in images] + [{"type": "text", "text": question + COT}]}]prompt = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)inputs = processor(text=[prompt], images=images, return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=1024, do_sample=False)print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Limitations

Small, noisy margins on two benchmarks. The MuirBench and Mantis gains are not statistically significant. Mantis-Eval has only 217 items.
Abstention is still weaker than base on image-option questions. The model abstains less often than the base model on MuirBench's unanswerable items, especially when the options are images. It sometimes answers "Neither" on two-option questions that offer no such choice.
Possible perception errors inherited from the teachers. The rationales come from GPT teachers. A blinded visual audit flagged about 15% of the training traces for at least one wrong per-image statement; these traces were kept in the SFT data, and the dataset marks them.
Research use only. The base model's license (Apache-2.0) applies to the weights. The training data inherits the terms of its source datasets.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Kaylee0501

Model Tree

Base

Qwen/Qwen3-VL-8B-Instruct

Fine-tuned

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

How it was built

Supervised fine-tuning.
- Data: 36,495 chain-of-thought traces over multi-image questions (see the dataset card).
- Traces were generated by GPT teachers without access to the gold answer. A trace was kept only if the teacher's first answer was correct.
- What was trained: full fine-tuning of the language model and the vision-language merger. The vision encoder was frozen.
- Settings: 1 epoch, learning rate 1e-5 with a cosine schedule, effective batch size 16, bf16 autocast over fp32 master weights.
- Images were capped at 1280·28·28 pixels each, at both training and evaluation time.
Weight interpolation. The final weights are θ = 0.3·θ_SFT + 0.7·θ_base.
- The SFT model improved multi-image skills but weakened some base skills, for example abstaining on unanswerable items and some single-image perception. Interpolation keeps most of the gain and recovers those skills.
- The value α = 0.3 was chosen from {0.3, 0.5, 0.7, 1.0} on a development split only: a stratified 25% of the evaluation items.
- The selection rule was fixed in advance: pick the candidate that beats the base model on the most benchmarks, breaking ties by mean gain.

Results

The same harness, prompt and greedy decoding were used for every model. Scoring is letter- or exact-match, with no LLM judge.

Held-out 75% of items (never used for any choice; this is the unbiased comparison):

Table with columns: Benchmark, Qwen3-VL-8B-Instruct, this model, Δ, exact McNemar p
Benchmark	Qwen3-VL-8B-Instruct	this model	Δ	exact McNemar p
BLINK (val)	63.7	67.6	+3.9	0.001
MIRB (multi-image subsets)	54.2	59.5	+5.3	0.001
MuirBench	75.1	76.1	+1.0	0.28

Full benchmarks. These include the 25% development items, so they are slightly optimistic:

Table with columns: BLINK, MuirBench, MIRB, Mantis
	BLINK	MuirBench	MIRB	Mantis
Qwen3-VL-8B-Instruct	63.6	73.5	55.2	77.0
SFT only (α = 1.0)	63.4	72.5	61.1	79.7
this model (α = 0.3)	67.3	74.4	60.0

Evaluation sets:

BLINK val: 1,901 items
MuirBench: 2,600 items
MIRB: 814 items across the multi-image subsets 3d_scene, analogy, attribute, count, food, image_jigsaw, sightseeing and visual_chain
Mantis-Eval: 217 items

Decontamination: every training sample containing an image near-identical to any evaluation image was removed before training (dHash, then pixel check).

Prompt format (use it for best results)

System prompt:

text

You are a careful visual reasoning assistant. Reason across all the images, grounding each observation in a specific image by index, then give the final answer inside <answer></answer> tags.

You are a careful visual reasoning assistant. Reason across all the images, grounding each observation in a specific image by index, then give the final answer inside <answer></answer> tags.

User turn:

All images first.
Then the question, with options written as A. … lines.
Then this instruction:

text

Reason step by step in this exact format:Understand: state what the question asks and what form the answer must take.Perceive: one line per image, "- Image k: ...", giving only the evidence in that image which bears on the question.Connect: say how the evidence across specific images combines or conflicts, citing at least two image indices. If you pick one image, name the closest rival and why it fails.Conclude: one sentence.Then give the final answer alone inside <answer></answer> tags, with no explanation and no placeholder text inside the tags.

Reason step by step in this exact format:Understand: state what the question asks and what form the answer must take.Perceive: one line per image, "- Image k: ...", giving only the evidence in that image which bears on the question.Connect: say how the evidence across specific images combines or conflicts, citing at least two image indices. If you pick one image, name the closest rival and why it fails.Conclude: one sentence.Then give the final answer alone inside <answer></answer> tags, with no explanation and no placeholder text inside the tags.

python

from transformers import AutoModelForImageTextToText, AutoProcessorfrom PIL import Image model_id = "Kaylee0501/Qwen3-VL-8B-CrossImage-SFT"model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="bfloat16", device_map="auto")processor = AutoProcessor.from_pretrained(model_id) SYSTEM = "You are a careful visual reasoning assistant. Reason across all the images, grounding each observation in a specific image by index, then give the final answer inside <answer></answer> tags."COT = "\n\nReason step by step in this exact format:\nUnderstand: ...\n..."   # paste the instruction above images = [Image.open("a.jpg"), Image.open("b.jpg")]question = "Do Image 1 and Image 2 show the same building?\nA. Yes\nB. No"messages = [{"role": "system", "content": [{"type": "text", "text": SYSTEM}]},            {"role": "user", "content": [{"type": "image"} for _ in images] + [{"type": "text", "text": question + COT}]}]prompt = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)inputs = processor(text=[prompt], images=images, return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=1024, do_sample=False)print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Limitations

Small, noisy margins on two benchmarks. The MuirBench and Mantis gains are not statistically significant. Mantis-Eval has only 217 items.
Abstention is still weaker than base on image-option questions. The model abstains less often than the base model on MuirBench's unanswerable items, especially when the options are images. It sometimes answers "Neither" on two-option questions that offer no such choice.
Possible perception errors inherited from the teachers. The rationales come from GPT teachers. A blinded visual audit flagged about 15% of the training traces for at least one wrong per-image statement; these traces were kept in the SFT data, and the dataset marks them.
Research use only. The base model's license (Apache-2.0) applies to the weights. The training data inherits the terms of its source datasets.