Results
Test split: 126 documents (100 CORD receipts, 26 invoices). Evaluated with this adapter loaded
from the Hub, greedy decoding, batch 1, and images capped at 1280×1280 px.
Table with columns: Model, JSON valid, Field F1, Value F1, Exact match, Lenient F1, Lenient EM| Model | JSON valid | Field F1 | Value F1 | Exact match | Lenient F1 | Lenient EM |
|---|
| Qwen3.5-2B zero-shot | 96.8% | 0.000 | 0.462 | 0.0% | 0.000 | 0.0% |
| + this adapter | 100% | 0.886 | 0.916 | 38.9% | 0.921 | 48.4% |
Table with columns: Source, n, Field F1, Exact match, Lenient F1, Lenient EM| Source | n | Field F1 | Exact match | Lenient F1 | Lenient EM |
|---|
| CORD receipts | 100 | 0.869 | 48.0% | 0.892 | 52.0% |
| Invoices | 26 | 0.910 | 3.8% | 0.964 | 34.6% |
- Field F1: precision/recall over
(path, value) leaves, with list order ignored, after
lower-casing and collapsing whitespace.
- Value F1: the same comparison on values alone. The zero-shot model reads the text but invents
its own keys, so it scores 0 on Field F1.
- Lenient: also ignores whitespace inside values and treats an object the same as a one-item list.
These are label artifacts, not reading errors: the invoice labels write
$ 978,12, while the model
writes $978,12.
- Exact match: the whole document is correct.
Usage
The prompt must match training. The only difference between the two document types is one word
(receipt / invoice). Tested with transformers 5.23.0, peft 0.21.2, and torch 2.14.1 on an L4 (24 GB).
import json, torchfrom PIL import Imagefrom peft import PeftModelfrom transformers import AutoModelForImageTextToText, AutoProcessor base = "Qwen/Qwen3.5-2B"model = AutoModelForImageTextToText.from_pretrained(base, dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(model, "zackyalgiffari/qwen3.5-2b-receipt-invoice-extract-lora").merge_and_unload()processor = AutoProcessor.from_pretrained(base) image = Image.open("receipt.jpg").convert("RGB")max_pixels = 1280 * 1280 # the training cap; larger images were downscaled, aspect keptif image.width * image.height > max_pixels: s = (max_pixels / (image.width * image.height)) ** 0.5 image = image.resize((int(image.width * s), int(image.height * s)), Image.BICUBIC) doc_type = "receipt" # or "invoice"messages = [{"role": "user", "content": [ {"type": "image", "image": image}, {"type": "text", "text": f"Extract all fields from this {doc_type} image. Respond with only a JSON object."},]}]prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = processor(text=[prompt], images=[[image]], return_tensors="pt").to(model.device)with torch.inference_mode(): out = model.generate(**inputs, max_new_tokens=1536, do_sample=False)print(json.loads(processor.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)))
With the GitHub repo installed, the equivalent command is
python -m invoice_vlm.infer --config configs/qwen35_2b_lora.yaml --adapter zackyalgiffari/qwen3.5-2b-receipt-invoice-extract-lora --type receipt receipt.jpg.
Training
- Data: CORD v2 train (800 receipts, Indonesian) plus invoices-donut-data-v1 train (425
invoices), interleaved. The target is each dataset's
gt_parse serialized as compact JSON.
- Method: LoRA (r=16, alpha=32, dropout 0.05) on every linear layer of the language model. The
vision tower is frozen. Loss is computed only on the answer tokens.
- Hyperparameters: 3 epochs, learning rate 1e-4 with cosine schedule and 3% warmup, effective
batch size 8 (1 × 8 accumulation), bf16, gradient checkpointing, seed 42.
- Compute: one Colab L4 (24 GB). Training took 2.5 h (462 steps) with a peak of 11.7 GiB.
Table with columns: Epoch, Step, Training loss, Validation loss| Epoch | Step | Training loss | Validation loss |
|---|
| 1 | 154 | 0.0514 | 0.0388 |
| 2 | 308 | 0.0324 | 0.0332 |
| 3 | 462 | 0.0134 | 0.0328 |
Limitations
- Two layouts only. The model learned two schemas: CORD's receipt keys (
menu[].nm,
total.total_price, ...) and the invoice dataset's keys (header, items[], summary). Expect
those keys, not a general invoice schema. Other layouts and languages are untested.
- Small invoice set. The invoice test split has only 26 documents, so its scores are noisy.
For example, invoice exact match is 3.8% at batch 1 and 7.7% at batch 4, a difference of one
document.
- Single characters slip. Remaining errors include dropped or swapped characters in long codes
(IBANs), item names, and totals. Check amounts before using them for anything that matters.
Licenses
The adapter is Apache-2.0, like the base model. The training data is
CORD v2 (CC-BY-4.0) and
invoices-donut-data-v1 (MIT,
by Katana ML).
Citation
If you use this adapter, please cite the CORD dataset it was trained on:
@inproceedings{park2019cord, title = {CORD: A Consolidated Receipt Dataset for Post-OCR Parsing}, author = {Park, Seunghyun and Shin, Seung and Lee, Bado and Lee, Junyeop and Surh, Jaeheung and Seo, Minjoon and Lee, Hwalsuk}, booktitle = {Workshop on Document Intelligence at NeurIPS 2019}, year = {2019}}