GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
gavinzsmeng
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
gavinzsmeng
Model Tree
Qwen/Qwen3-VL-8B-Instruct
Input Modalities
Output Modalities
Supported Functionality
The base model collapses onto the majority class on True/False questions: its recall on the minority answer ("No") is 39.7% — below random chance (50%). We fine-tuned to see whether standard SFT fixes this, and measured it with per-sample paired tests rather than raw accuracy alone.
Evaluated on 2,889 held-out test questions (split by source report, zero overlap with training), same prompt as training, greedy decoding.
| Condition | MC raw | TF raw | TF macro-recall | Recall @ "No" (95% Wilson CI) |
|---|---|---|---|---|
| Qwen3-VL-8B (base) | 73.70% | 75.23% | 63.73% | 39.7% [31.2, 48.8] |
| + this adapter | 84.13% | 84.01% | 79.15% | 69.0% [60.1, 76.7] |
Paired per-sample comparison (identical global uid):
markdown
base -> this adapter fixed 430 broke 136 net +294 McNemar p < 0.001
The gain concentrates almost entirely on the minority class:
markdown
"Yes" recall (n=328) 87.8% -> 89.3% +1.5pp"No" recall (n=116) 39.7% -> 69.0% +29.3pp (20x difference)
The confidence intervals for "No" recall ([31.2, 48.8] vs [60.1, 76.7]) do not overlap.
| Base model | Qwen/Qwen3-VL-8B-Instruct |
| Method | LoRA, r=16, α=32, target_modules=all-linear |
| Frozen | Vision encoder and aligner (freeze_vit=true) |
| Trainable params | 43.6M (0.50% of 8.81B) |
| Data | 13,465 questions / 2,322 reports (report-level split, 0 leakage) |
| Epochs | 2 (422 steps) |
| Effective batch | 1 × 16 grad-accum × 4 GPUs |
python
from transformers import AutoProcessor, AutoModelForImageTextToTextfrom peft import PeftModelREPO = "gavinzsmeng/visfineval-qwen3vl-8b-lora-natural"processor = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-8B-Instruct")base = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3-VL-8B-Instruct", dtype="bfloat16", device_map="cuda")model = PeftModel.from_pretrained(base, REPO).merge_and_unload().eval()from PIL import Imageimg = Image.open("chart.jpg").convert("RGB")PROMPT = ("请仔细阅读图片，回答问题，只输出正确选项的字母（A/B/C/D），不要解释。\n\n""问题：{q}\nA. {a}\nB. {b}\nC. {c}\nD. {d}\n\n答案：")msgs = [{"role": "user", "content": [{"type": "image", "image": img},{"type": "text", "text": PROMPT.format(q="...", a="...", b="...", c="...", d="...")},]}]inputs = processor.apply_chat_template(msgs, add_generation_prompt=True,tokenize=True, return_dict=True,return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=8, do_sample=False)print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])
⚠ Use the prompt above verbatim. Prompt wording moves minority-class recall by up to 23 points on this task (see Finding 4 in the audit), so a different phrasing will not reproduce these numbers.
For True/False questions, the prompt is:
markdown
请仔细阅读图片，回答问题，只回答「是」或「否」，不要解释。问题：{question}答案：
Training images come from Chinese broker research reports and are not redistributed
here. Obtain VisFinEval from its official channels:
SUFE-AIFLM-Lab/VisFinEval.
This adapter is a derivative of Qwen/Qwen3-VL-8B-Instruct (Apache-2.0) and is released
under Apache-2.0.
bibtex
@misc{meng2026visfinevalrel,title = {VisFinEval Reliability Audit: what reported accuracy hides},author = {Meng, Gavin C.},year = {2026},url = {https://github.com/gavinzsmeng/visfineval-reliability}}
| LR | 1e-4, cosine, 5% warmup |
| Precision | bfloat16 |
| Framework | ms-swift 4.6.0.dev0 |
| Hardware | 4× RTX 4090, DDP, ~1h36m |