qwen3.5-0.8b-slake-medvqa-lora API & Inference Endpoint

Model

Table

Base model	`Qwen/Qwen3.5-0.8B` @ `2fc06364715b967f1860aea9cf38778875588b17` (`Qwen3_5ForConditionalGeneration`)
Architecture	ViT (12 blocks, patch 16, 2x2 merge) + 24-layer LM (18 Gated DeltaNet + 6 gated full-attention)
Adapter	LoRA r=8, alpha=16, dropout=0.05
Target modules	LM `self_attn.{q,k,v,o}_proj` (6 layers), `linear_attn.{in_proj_qkv,in_proj_z,out_proj}` (18 layers)
Trainable params	2,457,600 / 855,443,520 (0.2873%)
Frozen	vision tower, merger, MLPs, embeddings, lm_head

Data

BoKelvin/SLAKE @ a9083ce6c34ac3ffb17671a605962924d8a8f9e9, both languages, no translation. Split mode image_disjoint: the official test split is used unchanged; train/validation are rebuilt from official train+validation after removing every image that appears in test (official SLAKE splits share images), grouped by image hash, stratified by modality.

Table with columns: split, QA pairs, images
split	QA pairs	images
train	8542	391
validation	1549	71
test	2094	180

Excluded records: empty_answer=1, duplicate_record=14, image_shared_with_test=1828.

Training

Table

Objective	answer-only causal LM loss (prompt and image tokens masked)
Batch	1 x grad. accum. 8 = 8
Optimizer	AdamW, lr 0.0001, weight decay 0.01, cosine schedule, warmup 0.03, grad clip 1.0
Epochs (max)	3 (early stopping patience 4 evals)
Precision	bf16 autocast, gradient checkpointing = True
Checkpoint selection	`eval_gen_exact_match` on validation
Image budget	65536-262144 px (<= 256 visual tokens)
Seed

Evaluation

Official SLAKE test split (2094 QA, greedy decoding, max 32 new tokens). Accuracy = normalized exact match or numeric match; normalization = NFKC, case-fold, whitespace collapse, outer punctuation strip.

Table with columns: subset, n, accuracy, exact match, token F1, yes/no acc.
subset	n	accuracy	exact match	token F1	yes/no acc.
overall	2094	77.2	77.2	84.1	87.9
answer_type=CLOSED	836	83.0	83.0	88.3	87.9
answer_type=OPEN	1258	73.4

Test answer-token loss: 0.456; empty-answer rate 0.0%; invalid-output rate 0.0%.

Image ablation (official test)

Table with columns: model / condition, accuracy, token F1
model / condition	accuracy	token F1
fine-tuned/correct	77.2	84.1
fine-tuned/none	30.8	40.9
fine-tuned/shuffled	39.6	49.4
base/correct	0.0	7.2
base/none	0.0	2.3
base/shuffled	0.1

Usage

python

import torchfrom PIL import Imagefrom peft import PeftModelfrom transformers import AutoModelForImageTextToText, AutoProcessor base = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.5-0.8B", dtype=torch.bfloat16).cuda()model = PeftModel.from_pretrained(base, "thealper2/qwen3.5-0.8b-slake-medvqa-lora").eval()processor = AutoProcessor.from_pretrained("thealper2/qwen3.5-0.8b-slake-medvqa-lora") question = "Does the picture contain liver?"messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "Answer the following question about this image:\n" + question}]}]prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)inputs = processor(text=[prompt], images=[Image.open("image.jpg").convert("RGB")], return_tensors="pt").to("cuda")out = model.generate(**inputs, max_new_tokens=32, do_sample=False)print(processor.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip())

Processor pixel budget used in training: size={'shortest_edge': 65536, 'longest_edge': 262144} (saved with the adapter).

Dataset attribution

SLAKE (Liu et al., ISBI 2021), CC BY 4.0: https://www.med-vqa.com/slake/

Base model

Qwen/Qwen3.5-0.8B @ 2fc06364715b967f1860aea9cf38778875588b17 (Qwen3_5ForConditionalGeneration)

Architecture

ViT (12 blocks, patch 16, 2x2 merge) + 24-layer LM (18 Gated DeltaNet + 6 gated full-attention)

Adapter

LoRA r=8, alpha=16, dropout=0.05

Target modules

LM self_attn.{q,k,v,o}_proj (6 layers), linear_attn.{in_proj_qkv,in_proj_z,out_proj} (18 layers)

Trainable params

2,457,600 / 855,443,520 (0.2873%)

Frozen

vision tower, merger, MLPs, embeddings, lm_head

split

QA pairs

images

train

8542

391

validation

1549

test

2094

180

Objective

answer-only causal LM loss (prompt and image tokens masked)

Batch

1 x grad. accum. 8 = 8

Optimizer

AdamW, lr 0.0001, weight decay 0.01, cosine schedule, warmup 0.03, grad clip 1.0

Epochs (max)

3 (early stopping patience 4 evals)

Precision

bf16 autocast, gradient checkpointing = True

Checkpoint selection

eval_gen_exact_match on validation

Image budget

65536-262144 px (<= 256 visual tokens)

Seed

subset

accuracy

exact match

token F1

yes/no acc.

overall

2094

77.2

84.1

87.9

answer_type=CLOSED

836

83.0

88.3

87.9

answer_type=OPEN

1258

73.4

model / condition

accuracy

token F1

fine-tuned/correct

77.2

84.1

fine-tuned/none

30.8

40.9

fine-tuned/shuffled

39.6

49.4

base/correct

0.0

7.2

base/none

0.0

2.3

base/shuffled

0.1

import torchfrom PIL import Imagefrom peft import PeftModelfrom transformers import AutoModelForImageTextToText, AutoProcessor base = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.5-0.8B", dtype=torch.bfloat16).cuda()model = PeftModel.from_pretrained(base, "thealper2/qwen3.5-0.8b-slake-medvqa-lora").eval()processor = AutoProcessor.from_pretrained("thealper2/qwen3.5-0.8b-slake-medvqa-lora") question = "Does the picture contain liver?"messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "Answer the following question about this image:\n" + question}]}]prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)inputs = processor(text=[prompt], images=[Image.open("image.jpg").convert("RGB")], return_tensors="pt").to("cuda")out = model.generate(**inputs, max_new_tokens=32, do_sample=False)print(processor.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip())

qwen3.5-0.8b-slake-medvqa-lora

README

Model

Data

Training

Evaluation

Image ablation (official test)

Usage

Dataset attribution

Explore FriendliAI today

README

Model

Data

Training

Evaluation

Image ablation (official test)

Usage

Dataset attribution