AnoushkaBaidya

qwen2vl-product-attribute-qlora

README

License: apache-2.0

Key results

Paired evaluation on 2,500 held-out examples:

Table with columns: Metric, Base, QLoRA, Change
Metric	Base	QLoRA	Change
Exact match	21.68%	70.40%	+48.72 pp
Macro F1	55.04%	85.31%	+30.27 pp
Micro F1	58.83%	87.33%	+28.50 pp
Strict JSON	15.96%	99.84%	+83.88 pp

1,246 QLoRA wins vs. 28 regressions
1.69× two-GPU throughput · 84.6% scaling efficiency

The repository contains the PEFT adapter only; the Qwen2-VL base model is loaded separately.

Model overview

Table with columns: Property, Value
Property	Value
Base model	Qwen/Qwen2-VL-2B-Instruct
Adaptation	QLoRA
Task	Multimodal product attribute extraction
Training examples	20,000
Validation examples	500
Training optimizer steps	2,500
Training hardware	2 × NVIDIA GeForce RTX 4090
Precision	bfloat16 compute
Quantization

Evaluation objective

This project was designed as an end-to-end multimodal ML systems.

Evaluation covers both predictive quality and operational characteristics of the multimodal system.

The evaluation framework measures:

attribute-level predictive performance,
exact structured-output correctness,
JSON reliability,
paired improvement relative to the unmodified base model,
deterministic reproducibility,
inference latency,
and distributed-training scalability.

The evaluation characterizes the adapted model across the complete inference workflow, including structured-output reliability and computational performance in addition to attribute-level predictive quality.

The reported results correspond to the documented training configuration and held-out evaluation protocol. No model ensembling or task-specific post-processing was applied to the reported predictions.

Held-out evaluation

Evaluation used a deterministic, seeded 2,500-example subset of the independent 14,293-example test split. The base model and QLoRA model were evaluated on the same examples.

Table with columns: Metric, Base model, QLoRA, Change
Metric	Base model	QLoRA	Change
Normalized exact match	21.68%	70.40%	+48.72 pp
Macro attribute accuracy	52.76%	82.31%	+29.55 pp
Macro attribute F1	55.04%	85.31%	+30.27 pp
Micro F1	58.83%

Paired outcomes

Across the 2,500 paired examples:

Table with columns: Outcome, Count
Outcome	Count
QLoRA win	1,246
QLoRA regression	28
Both correct	514
Both incorrect	712

The normalized exact-match improvement was +48.72 percentage points, with a 95% bootstrap confidence interval of +46.64 to +50.72 percentage points (2,000 bootstrap samples, seed 42). Mean per-example labeled-attribute accuracy improved by +22.70 percentage points, with a 95% bootstrap confidence interval of +21.86 to +23.50 percentage points. Schema validity was already effectively saturated for the base model under the evaluation parser. The more meaningful structured-output change was strict JSON compliance, which improved from 15.96% to 99.84%.

Distributed training

Training was scaled from one to two RTX 4090 GPUs using PyTorch distributed training through Hugging Face Accelerate.

At a matched effective global batch size of 8:

Table with columns: Configuration, Throughput
Configuration	Throughput
1 × RTX 4090	1.128 examples/s
2 × RTX 4090	1.909 examples/s

This corresponds to:

1.69× throughput speedup
84.6% two-GPU scaling efficiency

These values are measured rather than theoretical scaling claims.

Output schema

The model is trained to return JSON in the following form:

json
{
  "product_type": "CELLULAR_PHONE_CASE",
  "brand": "Example Brand",
  "color": ["Black"],
  "material": ["Plastic"]
}

Missing multi-valued attributes are represented by empty lists.

Usage

Install the required libraries:

bash
pip install torch transformers peft bitsandbytes accelerate qwen-vl-utils pillow

Load the base model and adapter:

python
import torch
from peft import PeftModel
from transformers import (
    AutoProcessor,
    BitsAndBytesConfig,
    Qwen2VLForConditionalGeneration,
)

BASE_MODEL_ID = "Qwen/Qwen2-VL-2B-Instruct"
ADAPTER_ID = "AnoushkaBaidya/qwen2vl-product-attribute-qlora"

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

processor = AutoProcessor.from_pretrained(
    ADAPTER_ID,
    use_fast=False,
)

base_model = Qwen2VLForConditionalGeneration.from_pretrained(
    BASE_MODEL_ID,
    quantization_config=quantization_config,
    dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="sdpa",
)

model = PeftModel.from_pretrained(
    base_model,
    ADAPTER_ID,
    is_trainable=False,
)

model.eval()

The associated GitHub project provides the complete multimodal inference path, prompt construction, API service, evaluation pipeline, Docker packaging, and training workflow.

Training data

Training data was derived from the Amazon Berkeley Objects (ABO) dataset.

The project pipeline performs:

raw dataset acquisition and inventory,
listing normalization,
image metadata construction,
listing-image alignment,
image preprocessing,
multimodal manifest construction,
deterministic train/validation/test splitting,
instruction-data generation,
integrity and overlap validation,
QLoRA fine-tuning,
held-out evaluation.

The public GitHub repository contains the reproducible pipeline definitions and validation code. Dataset images are not redistributed with this adapter.

Reproducibility

The reported training run used:

base model: Qwen/Qwen2-VL-2B-Instruct
20,000 training examples
validation subset: 500 examples
2,500 optimizer steps
2 × RTX 4090
effective global batch size: 8
QLoRA rank: 16
QLoRA alpha: 32
LoRA dropout: 0.05
NF4 4-bit quantization
double quantization enabled
bfloat16 compute
maximum sequence length: 2,048
deterministic held-out evaluation subset seed: 42

A full sequence-length audit over the 20,000-example training subset observed a maximum sequence length of 1,590 tokens, below the configured 2,048-token limit.

Inference characteristics

The fine-tuned model substantially improves structured extraction quality, but in the measured evaluation environment it also increases generation latency relative to the base model.

For the 2,500-example benchmark:

Table with columns: Model, Mean generation latency
Model	Mean generation latency
Base	2,060 ms/example
QLoRA	3,091 ms/example

Latency depends strongly on GPU architecture, software versions, quantization backend, image resolution, and serving configuration. These measurements should therefore be treated as environment-specific rather than universal serving latency guarantees.

Tested environments

Training and benchmark environment

Python 3.11
PyTorch 2.7.1 + CUDA 12.6
Transformers 4.53.2
PEFT 0.19.1
Accelerate 1.8.1
bitsandbytes 0.46.1

Independent local serving validation

The published adapter was also loaded and served successfully on an NVIDIA GeForce RTX 5070 Ti Laptop GPU using:

PyTorch 2.12.1 + CUDA 13.0
Transformers 4.57.6
PEFT 0.19.1
Accelerate 1.14.0
bitsandbytes 0.49.2

This second environment is reported as a compatibility check rather than a benchmark-equivalent runtime.

Limitations

The adapter targets four catalog attributes and is not a general-purpose product understanding model.
Product metadata may be multilingual, duplicated, incomplete, or noisy.
Some source labels contain multiple aliases or missing attributes.
The reported benchmark uses a deterministic 2,500-example subset of the larger held-out test split rather than the entire test partition.
Training was performed on a deterministic 20,000-example subset of the prepared training partition.
The project prioritizes end-to-end ML engineering, reproducibility, deployment, and measured improvement over leaderboard-oriented optimization.
Predictions should not be treated as authoritative catalog data without downstream validation in high-consequence workflows.

Intended use

This adapter is intended for:

product catalog enrichment,
multimodal metadata extraction,
catalog consistency workflows,
ML engineering experimentation,
PEFT/QLoRA research and education.

It is not intended for safety-critical or high-consequence automated decisions.

License

The base model, Qwen/Qwen2-VL-2B-Instruct, is distributed under the Apache License 2.0.

This repository contains adapter weights and supporting processor configuration for use with that base model. Users are responsible for complying with the licenses and terms applicable to the base model and any datasets used with it.

Project

The full engineering project includes:

DVC-based data pipelines,
multimodal dataset validation,
image preprocessing,
QLoRA training,
PyTorch distributed multi-GPU training,
Hugging Face Accelerate,
MLflow experiment tracking,
paired held-out evaluation,
inference benchmarking,
FastAPI serving,
interactive browser inference,
Docker packaging,
and CI/CD.

GitHub:

https://github.com/AnoushkaBaidya/multimodal-product-intelligence

Artifact integrity

Final adapter SHA-256:

4f564becbfa204da87ef9605c1d34bb5dc8612accf2220ab9fc8eec8120a52c0

The published adapter was verified against the final exported training artifact before Hub upload.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.