dmnsjk

qwen2.5-vl-7b-cord-json-lora

README

License: apache-2.0

Usage

python
import torch
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
from peft import PeftModel

base = "Qwen/Qwen2.5-VL-7B-Instruct"
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(base, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "dmnsjk/qwen2.5-vl-7b-cord-json-lora")
processor = AutoProcessor.from_pretrained("dmnsjk/qwen2.5-vl-7b-cord-json-lora")

Training configuration

Table

base model	`Qwen/Qwen2.5-VL-7B-Instruct`
LoRA rank / alpha / dropout	32 / 64 / 0.05
target modules	`q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj`
vision tower	frozen, not adapted
dataset	`naver-clova-ix/cord-v2`
resolution cap	`min_pixels=200704`, `max_pixels=1003520`
epochs / lr / scheduler	3.0 / 0.0001 / cosine
precision	bf16, gradient checkpointing, `loss_type="chunked_nll"`

run_config.yaml in this repository is the exact resolved configuration of the run.

Evaluation

Held-out CORD-v2 test split, 100 receipts, greedy decoding. Scored against the base checkpoint under an identical prompt, so the delta is attributable to the adapter.

Table with columns: Metric, Base (no adapter), This adapter
Metric	Base (no adapter)	This adapter
JSON parse rate (strict)	0.030	1.000
JSON parse rate (lenient)	0.990	1.000
Field precision	0.288	0.860
Field recall	0.651	0.839
Field F1	0.400	0.849
Exact match

JSON parse-rate is reported separately from field accuracy on purpose: a model that extracts every field but emits a trailing comma scores zero downstream. "Strict" is json.loads on the raw generation; "lenient" first strips a markdown code fence.

Limitations

Small, narrow training set. 800 training examples, 3 epochs. CORD-v2 is Indonesian restaurant/retail receipts specifically. Do not expect this to transfer to invoices, financial tables, or engineering schematics without retraining — the brief's broader document types were never trained on.
Exact match is 0.42. More than half of outputs differ from the gold parse somewhere, usually a single OCR-level field (a digit in a price, an abbreviated item name). Normalised edit similarity is 0.946, so errors are typically small rather than structural — but this is not a solved extraction task.
The schema is fixed by the system prompt this adapter was trained with (see run_config.yaml). Prompting it for a different key set will degrade it. The adapter has been specialised toward one output shape.
Field scoring ignores list order (receipt line items are treated as an unordered bag), so ordering mistakes are not penalised.
Resolution is capped at max_pixels (see the table above). Documents far denser than a receipt may lose legibility at that cap.

Source code

Training pipeline, SLURM sbatch scripts and Accelerate/FSDP configs: https://github.com/dmn-sjk/vlm_doc_parsing.git

See src/infer/predict.py there for a complete inference example, including --merge (merge_and_unload) for faster serving.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

dmnsjk

Model Tree

Base

Qwen/Qwen2.5-VL-7B-Instruct

Adapter

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer