GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
gavinzsmeng
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
gavinzsmeng
Model Tree
Qwen/Qwen3-VL-8B-Instruct
Input Modalities
Output Modalities
Supported Functionality
This is the ablation arm. The base model collapses onto the majority class on True/False questions ("No" recall 39.7%, below chance). The obvious hypothesis is imbalanced training data. This adapter tests it.
The hypothesis is wrong. Class balancing gave no benefit.
Same held-out test split (2,889 questions, report-level split, zero overlap), same prompt, greedy decoding.
| Condition | MC raw | TF raw | TF macro-recall | Recall @ "No" (95% Wilson CI) |
|---|---|---|---|---|
| Qwen3-VL-8B (base) | 73.70% | 75.23% | 63.73% | 39.7% [31.2, 48.8] |
| LoRA natural (sibling) | 84.13% | 84.01% | 79.15% | 69.0% [60.1, 76.7] |
| this adapter (balanced) | 82.78% | 84.01% | 79.43% | 69.8% [60.9, 77.4] |
Paired per-sample comparison against the natural adapter:
markdown
natural -> balanced fixed 84 broke 117 net -33 McNemar p = 0.024
The minority-class collapse was not caused by imbalanced training data. Standard SFT on the natural distribution already fixes it — the improvement appears to come from the model learning the task (reading the chart) rather than from seeing more minority examples. That mechanism is a hypothesis, not something this experiment proves.
The practical takeaway: before resampling a skewed dataset, check whether plain SFT already solves the problem. Here it did.
Identical to the natural adapter except for the data:
| Base model | Qwen/Qwen3-VL-8B-Instruct |
| Method | LoRA, r=16, α=32, target_modules=all-linear |
| Frozen | Vision encoder and aligner (freeze_vit=true) |
| Trainable params | 43.6M (0.50% of 8.81B) |
| Data | 40,350 rows (6 × 6,725), from the same 2,322 training reports |
| Epochs | 2 (1,262 steps) |
| Effective batch | 1 × 16 grad-accum × 4 GPUs |
Identical to the sibling adapter — see
visfineval-qwen3vl-8b-lora-natural
for the full snippet and the exact prompts.
⚠ Use the documented prompt verbatim. Prompt wording moves minority-class recall by up to 23 points on this task; a different phrasing will not reproduce these numbers.
Same as the natural adapter, plus:
Training images come from Chinese broker research reports and are not redistributed.
Obtain VisFinEval from
SUFE-AIFLM-Lab/VisFinEval.
Derivative of Qwen/Qwen3-VL-8B-Instruct (Apache-2.0), released under Apache-2.0.
bibtex
@misc{meng2026visfinevalrel,title = {VisFinEval Reliability Audit: what reported accuracy hides},author = {Meng, Gavin C.},year = {2026},url = {https://github.com/gavinzsmeng/visfineval-reliability}}
| LR | 1e-4, cosine, 5% warmup |
| Precision | bfloat16 |
| Framework | ms-swift 4.6.0.dev0 |
| Hardware | 4× RTX 4090, DDP, ~4h55m |