prodigalbeef

qwen3vl-chartqa-qlora

README

License: apache-2.0

Results

Accuracy by answer type

Table with columns: Relaxed accuracy
	Relaxed accuracy
Base (4-bit, zero-shot)	0.838 (419/500)
QLoRA fine-tuned	0.856 (428/500)
Delta	+1.8 pts

Table with columns: Answer type, n, Base, Fine-tuned, Δ
Answer type	n	Base	Fine-tuned	Δ
Numeric	382	0.859	0.872	+1.3
Text	90	0.789	0.833	+4.4
Yes/No	28	0.714	0.714	0.0
Human-written questions	245	0.743	0.759	+1.6
Machine-generated questions	255	0.929	0.949	+2.0

Error patterns. The errors that shrank most are format mismatches, which fell from 6 to 3. One example is answering "2011" when the gold answer is "March 2011". Large numeric misses also fell, from 30 to 26. Reasoning-heavy questions remain the main failure mode on human-written charts, such as counting bars above a threshold or averaging series. Full tables, including the error taxonomy and sample failures, are in results/report.md.

How solid is the +1.8? At the level of individual questions, fine-tuning fixed 24 answers and broke 15, for a net gain of 9. An exact McNemar test on those 39 flips gives p ≈ 0.20. The gain is consistent in direction across every category except yes/no, but on 500 questions it is not statistically significant. A full-test-set evaluation (2,500 questions) would settle it.

Training loss

The loss is computed on answer tokens only. Its mean fell from 0.69 over the first 20 steps to 0.35 over the last 20. The per-step loss is noisy because answers are only 1–3 tokens long. The plot shows 50-step means, also available in results/loss_binned.csv.

Method

text

ChartQA image + question   │  resize longest side ≤ 768 px · prompt: "Answer … with a single word or number."   ▼Qwen3-VL-4B-Instruct  (frozen, 4-bit NF4)   ├─ Vision encoder  ── LoRA on attn.qkv, attn.proj        (48 modules)   └─ Language model  ── LoRA on q_proj, k_proj, v_proj, o_proj (144 modules)   ▼SFT (TRL + Unsloth) · loss masked to the assistant answer only   ▼Greedy decode (≤24 tokens) → first line → ChartQA relaxed accuracy

Why both towers. ChartQA errors are often perception errors, such as misreading a bar height or an axis label, not just reasoning errors. Adapting only the LLM would leave the vision encoder's features fixed. The code asserts that LoRA is attached to both towers.
Memory on a T4. The setup uses 4-bit base weights, Unsloth gradient checkpointing with gradient offload, 8-bit AdamW and a 768 px image cap. Images are decoded lazily per batch, so the 28k-image train set never sits in RAM.
Metric. This is ChartQA's standard relaxed accuracy: numeric answers count as correct within 5% relative error, and everything else needs a case-insensitive exact match. It is implemented in chartqa_qlora/metrics.py and unit-tested.
Fair baseline. The same quantized model, prompt, image size, decoding settings and seeded 500-question subset are used before and after training.

Hyperparameters

Table

LoRA rank / α / dropout	16 / 16 / 0
Learning rate / schedule	2e-4, linear, 50 warmup steps
Batch	2 × 4 grad-accum = 8
Steps	1,900 (≈15.2k examples, 0.54 epoch)
Optimizer	AdamW 8-bit, weight decay 0.01
Precision	fp16 (T4 has no bf16)
Seed	3407

Quick start

Requirements: an NVIDIA GPU with at least 16 GB (a T4 works) and Python 3.11+.

bash

git clone https://huggingface.co/prodigalbeef/qwen3vl-chartqa-qloracd qwen3vl-chartqa-qlorapip install -r requirements.txt

Ask a question about a chart with the trained adapter (stored at the repo root):

bash

python scripts/infer.py --image my_chart.png --question "What was the value in 2019?"

Or load it with PEFT directly:

python

from unsloth import FastVisionModelmodel, tokenizer = FastVisionModel.from_pretrained("prodigalbeef/qwen3vl-chartqa-qlora", load_in_4bit=True)FastVisionModel.for_inference(model)

Reproduce the experiment (baseline eval → train → fine-tuned eval → outputs/report.md):

bash

python scripts/run_experiment.py --smoke     # ~15 min sanity checkpython scripts/run_experiment.py             # full run, ~3 h on one T4python scripts/run_experiment.py --max-side 640   # if you hit OOM

Re-score saved predictions on a CPU: python scripts/rescore.py baseline.jsonl finetuned.jsonl. Tests: pytest.

The original Kaggle notebook, with outputs stripped, is in notebooks/.

Project layout

text

chartqa_qlora/        core library  config.py           hyperparameters, prompt, LoRA target regex  data.py             ChartQA loading, image resize, chat formatting  model.py            4-bit loading, LoRA attachment (asserts both towers)  train.py            TRL SFT with answer-only loss + time-cap callback  evaluate.py         greedy decoding + relaxed accuracy, writes predictions  metrics.py          relaxed accuracy, answer types, error taxonomy  report.py           baseline vs fine-tuned Markdown reportscripts/              run_experiment.py · infer.py · rescore.pynotebooks/            original Kaggle notebookresults/              report.md (reported run) · loss_binned.csvassets/               figuresadapter_config.json   trained LoRA adapter (repo root, so PEFT/Unsloth load the repo id directly)adapter_model.safetensorstests/                metric unit tests

Limitations

Small eval. The evaluation uses 500 of 2,500 test questions. The overall gain is real in direction but not statistically significant (see above). Yes/no has only 28 questions.
Partial epoch. Training covered about 0.54 epoch because of the free-tier time budget. Longer training, or a higher rank, may help more.
Single seed. There is no variance estimate across seeds.
Strong base model. Qwen3-VL-4B already scores 83.8% zero-shot, so there is limited headroom. Most of the remaining errors are multi-step reasoning, which LoRA SFT on short answers does not target.
Short-answer style. The model is tuned to reply with a single word or number. Expect terse outputs and no explanations.

Acknowledgements

I am deeply grateful to my faculty mentor, Dr Rakesh Kumar Siddhartan, Amrita Vishwa Vidyapeetham, for his guidance and support throughout this project.

This project also builds on Qwen3-VL by the Qwen team, the ChartQA dataset (Masry et al., 2022), and the Unsloth, TRL, PEFT and bitsandbytes libraries. Compute was provided by Kaggle's free GPU tier.

License

The code is released under Apache-2.0, matching the base model's license. ChartQA is subject to its own terms.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

prodigalbeef

Model Tree

Base

Qwen/Qwen3-VL-4B-Instruct

Adapter

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Results

Accuracy by answer type

Table with columns: Relaxed accuracy
	Relaxed accuracy
Base (4-bit, zero-shot)	0.838 (419/500)
QLoRA fine-tuned	0.856 (428/500)
Delta	+1.8 pts

Table with columns: Answer type, n, Base, Fine-tuned, Δ
Answer type	n	Base	Fine-tuned	Δ
Numeric	382	0.859	0.872	+1.3
Text	90	0.789	0.833	+4.4
Yes/No	28	0.714	0.714	0.0
Human-written questions	245	0.743	0.759	+1.6
Machine-generated questions	255	0.929	0.949	+2.0

Training loss

Method

text

ChartQA image + question   │  resize longest side ≤ 768 px · prompt: "Answer … with a single word or number."   ▼Qwen3-VL-4B-Instruct  (frozen, 4-bit NF4)   ├─ Vision encoder  ── LoRA on attn.qkv, attn.proj        (48 modules)   └─ Language model  ── LoRA on q_proj, k_proj, v_proj, o_proj (144 modules)   ▼SFT (TRL + Unsloth) · loss masked to the assistant answer only   ▼Greedy decode (≤24 tokens) → first line → ChartQA relaxed accuracy

ChartQA image + question   │  resize longest side ≤ 768 px · prompt: "Answer … with a single word or number."   ▼Qwen3-VL-4B-Instruct  (frozen, 4-bit NF4)   ├─ Vision encoder  ── LoRA on attn.qkv, attn.proj        (48 modules)   └─ Language model  ── LoRA on q_proj, k_proj, v_proj, o_proj (144 modules)   ▼SFT (TRL + Unsloth) · loss masked to the assistant answer only   ▼Greedy decode (≤24 tokens) → first line → ChartQA relaxed accuracy

Why both towers. ChartQA errors are often perception errors, such as misreading a bar height or an axis label, not just reasoning errors. Adapting only the LLM would leave the vision encoder's features fixed. The code asserts that LoRA is attached to both towers.
Memory on a T4. The setup uses 4-bit base weights, Unsloth gradient checkpointing with gradient offload, 8-bit AdamW and a 768 px image cap. Images are decoded lazily per batch, so the 28k-image train set never sits in RAM.
Metric. This is ChartQA's standard relaxed accuracy: numeric answers count as correct within 5% relative error, and everything else needs a case-insensitive exact match. It is implemented in chartqa_qlora/metrics.py and unit-tested.
Fair baseline. The same quantized model, prompt, image size, decoding settings and seeded 500-question subset are used before and after training.

Hyperparameters

Table

LoRA rank / α / dropout	16 / 16 / 0
Learning rate / schedule	2e-4, linear, 50 warmup steps
Batch	2 × 4 grad-accum = 8
Steps	1,900 (≈15.2k examples, 0.54 epoch)
Optimizer	AdamW 8-bit, weight decay 0.01
Precision	fp16 (T4 has no bf16)
Seed	3407

Quick start

Requirements: an NVIDIA GPU with at least 16 GB (a T4 works) and Python 3.11+.

bash

git clone https://huggingface.co/prodigalbeef/qwen3vl-chartqa-qloracd qwen3vl-chartqa-qlorapip install -r requirements.txt

git clone https://huggingface.co/prodigalbeef/qwen3vl-chartqa-qloracd qwen3vl-chartqa-qlorapip install -r requirements.txt

Ask a question about a chart with the trained adapter (stored at the repo root):

bash

python scripts/infer.py --image my_chart.png --question "What was the value in 2019?"

python scripts/infer.py --image my_chart.png --question "What was the value in 2019?"

Or load it with PEFT directly:

python

from unsloth import FastVisionModelmodel, tokenizer = FastVisionModel.from_pretrained("prodigalbeef/qwen3vl-chartqa-qlora", load_in_4bit=True)FastVisionModel.for_inference(model)

from unsloth import FastVisionModelmodel, tokenizer = FastVisionModel.from_pretrained("prodigalbeef/qwen3vl-chartqa-qlora", load_in_4bit=True)FastVisionModel.for_inference(model)

Reproduce the experiment (baseline eval → train → fine-tuned eval → outputs/report.md):

bash

python scripts/run_experiment.py --smoke     # ~15 min sanity checkpython scripts/run_experiment.py             # full run, ~3 h on one T4python scripts/run_experiment.py --max-side 640   # if you hit OOM

python scripts/run_experiment.py --smoke     # ~15 min sanity checkpython scripts/run_experiment.py             # full run, ~3 h on one T4python scripts/run_experiment.py --max-side 640   # if you hit OOM

Re-score saved predictions on a CPU: python scripts/rescore.py baseline.jsonl finetuned.jsonl. Tests: pytest.

The original Kaggle notebook, with outputs stripped, is in notebooks/.

Project layout

text

chartqa_qlora/        core library  config.py           hyperparameters, prompt, LoRA target regex  data.py             ChartQA loading, image resize, chat formatting  model.py            4-bit loading, LoRA attachment (asserts both towers)  train.py            TRL SFT with answer-only loss + time-cap callback  evaluate.py         greedy decoding + relaxed accuracy, writes predictions  metrics.py          relaxed accuracy, answer types, error taxonomy  report.py           baseline vs fine-tuned Markdown reportscripts/              run_experiment.py · infer.py · rescore.pynotebooks/            original Kaggle notebookresults/              report.md (reported run) · loss_binned.csvassets/               figuresadapter_config.json   trained LoRA adapter (repo root, so PEFT/Unsloth load the repo id directly)adapter_model.safetensorstests/                metric unit tests

chartqa_qlora/        core library  config.py           hyperparameters, prompt, LoRA target regex  data.py             ChartQA loading, image resize, chat formatting  model.py            4-bit loading, LoRA attachment (asserts both towers)  train.py            TRL SFT with answer-only loss + time-cap callback  evaluate.py         greedy decoding + relaxed accuracy, writes predictions  metrics.py          relaxed accuracy, answer types, error taxonomy  report.py           baseline vs fine-tuned Markdown reportscripts/              run_experiment.py · infer.py · rescore.pynotebooks/            original Kaggle notebookresults/              report.md (reported run) · loss_binned.csvassets/               figuresadapter_config.json   trained LoRA adapter (repo root, so PEFT/Unsloth load the repo id directly)adapter_model.safetensorstests/                metric unit tests

Limitations

Small eval. The evaluation uses 500 of 2,500 test questions. The overall gain is real in direction but not statistically significant (see above). Yes/no has only 28 questions.
Partial epoch. Training covered about 0.54 epoch because of the free-tier time budget. Longer training, or a higher rank, may help more.
Single seed. There is no variance estimate across seeds.
Strong base model. Qwen3-VL-4B already scores 83.8% zero-shot, so there is limited headroom. Most of the remaining errors are multi-step reasoning, which LoRA SFT on short answers does not target.
Short-answer style. The model is tuned to reply with a single word or number. Expect terse outputs and no explanations.

Acknowledgements

I am deeply grateful to my faculty mentor, Dr Rakesh Kumar Siddhartan, Amrita Vishwa Vidyapeetham, for his guidance and support throughout this project.

This project also builds on Qwen3-VL by the Qwen team, the ChartQA dataset (Masry et al., 2022), and the Unsloth, TRL, PEFT and bitsandbytes libraries. Compute was provided by Kaggle's free GPU tier.

License

The code is released under Apache-2.0, matching the base model's license. ChartQA is subject to its own terms.