Results
Table with columns: Task, Metric, Baseline, Multi-task| Task | Metric | Baseline | Multi-task |
|---|
| LaTeX OCR | BLEU-4 | 0.8957 | 0.9533 |
| LaTeX OCR | CER | 0.1483 | 0.0543 |
| LaTeX OCR | Exact Match % | 0.0 | 54.0 |
| ChartQA | Exact Match % | 0.0 | 68.5 |
| ChartQA | Relaxed EM % | 0.0 | 66.5 |
Multi-task vs single-task cost (LaTeX EM): -0.5 pts
(95% paired-bootstrap CI comfortably includes 0 — not distinguishable from no cost at n=200 eval samples)
⚠️ Important caveat on the "0% baseline" numbers
The baseline exact-match scores above look dramatic (0.0% on both tasks) but this is largely
a formatting artifact, not zero underlying capability. The un-finetuned base model answers
conversationally (e.g. "Based on the chart... ✅ Answer: 63") rather than with a bare terse
string, so strict exact-match fails even when the content is correct. Under a lenient/normalized
match, baseline ChartQA accuracy is closer to ~78%, not 0%.
What fine-tuning appears to primarily provide is output-format discipline (terse, direct
answers) rather than new task knowledge — a LoRA adapter trained only on LaTeX OCR (never
seeing a single ChartQA example) scores ~51% exact-match on ChartQA, almost entirely by
unlocking latent chart-reading ability the base model already had. See assets/transfer_matrix.png
and assets/examples.md for supporting qualitative examples.
Eval set size note
Evaluation used 200 held-out examples per task. Bootstrap 95% CIs on exact-match
are ~13-14 points wide at this sample size, so small deltas (LoRA rank sweep, single- vs multi-task
cost, seed variance) should be read as noise-level, not as confirmed effects.
Training config
- Base model:
unsloth/Qwen3.5-0.8B
- LoRA rank: 32, alpha: 32
- Learning rate: 2e-4, batch size: 20, epochs: 1
- Train samples: 20000 LaTeX + 18000 ChartQA
- Training time: 23.6 min | Peak GPU: 21.686 GB (NVIDIA A100-SXM4-80GB)
See assets/ for loss curves, comparison plots, and qualitative best/worst prediction examples.