rukiyeberna

qwen2.5-vl-3b-food-extract-lora

README

License: apache-2.0

Model Details

Model Description

This repository contains the parameter-efficient LoRA adapter weights for Qwen2.5-VL-3B-Instruct, fine-tuned for multimodal structured attribute extraction from food photography. The model analyzes dish images and outputs strictly formatted JSON including food presence verification, title, ingredients, and accompanying beverages.

Developed by: Rukiye Berna Turan
Model type: Vision-Language Model (VLM) LoRA Adapter
Language(s) (NLP): English
License: Apache-2.0
Finetuned from model: Qwen/Qwen2.5-VL-3B-Instruct

Model Sources

Repository: https://github.com/rukiyeberna/vlm-food-extractor
Base Model Card: https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct

Uses

Direct Use

Zero-shot and structured visual extraction of culinary attributes from food images. Designed to produce strict JSON schemas for menu parsing, dietary tracking, recipe indexing, and food dataset labeling.

Out-of-Scope Use

Medical allergen confirmation or clinical dietary advice.
Non-food visual document parsing.
Precise weight, volume, or caloric estimation.

Bias, Risks, and Limitations

The model may miss occluded, mixed, or interior ingredients that are not visually apparent on the plate surface.
Exact token matching can vary compared to colloquial culinary names (e.g., predicting tomato salad instead of caprese salad).

How to Get Started with the Model

python
import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import PeftModel

BASE_MODEL = "Qwen/Qwen2.5-VL-3B-Instruct"
ADAPTER_PATH = "rukiyeberna/qwen2.5-vl-3b-food-extract-lora"

# Load base model & LoRA adapter
base_model = AutoModelForImageTextToText.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
model = PeftModel.from_pretrained(base_model, ADAPTER_PATH)
processor = AutoProcessor.from_pretrained(ADAPTER_PATH)
model.eval()

# Load image and build prompt
image = Image.open("sample_food.jpg").convert("RGB")
prompt = """Analyze this image and return only valid JSON with exactly these keys:
- is_food
- image_title
- food_items
- drink_items"""

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": prompt},
        ],
    }
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], padding=True, return_tensors="pt").to(model.device)

with torch.inference_mode():
    generated_ids = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
        repetition_penalty=1.15,
        no_repeat_ngram_size=3
    )

output = processor.batch_decode(
    generated_ids[:, inputs.input_ids.shape[1]:], 
    skip_special_tokens=True
)[0]

print(output)

Training Details

Training Data

Trained on mrdbourke/FoodExtract-1k-Vision (1,510 training samples) preprocessed into LLaVA-style conversational format (image + structured JSON response).

Training Hyperparameters

Regime: LoRA PEFT (Vision encoder and LLM backbone frozen)[cite: 1]
Precision: bfloat16[cite: 1]
LoRA Rank ( $r$ ): 32[cite: 1]
LoRA Alpha ( $α$ ): 64
LoRA Dropout: 0.05[cite: 1]
Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj[cite: 1]

Evaluation

Testing Data & Metrics

Evaluated on unseen food test images across structured schema integrity, sequence similarity, and fuzzy set-based overlap[cite: 1]:

JSON Parsing Success Rate[cite: 1]
is_food Classification Accuracy[cite: 1]
Title Character Similarity (SequenceMatcher)[cite: 1]
Set-based Item $F 1 $ Score (Precision / Recall)[cite: 1]
ROUGE-1 / ROUGE-L Overlap

Technical Specifications

Architecture

Base Architecture: Qwen2.5-VL (3B Parameters)[cite: 1]
Adapter Type: Low-Rank Adaptation (LoRA) on Self-Attention and MLP Projections[cite: 1]
Hardware: Single GPU (NVIDIA T4 / A100 environment)

Framework Versions

PEFT: 0.19.1
Transformers: 4.49.0+
PyTorch: 2.x
qwen-vl-utils: Latest

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

rukiyeberna

Model Tree

Base

Qwen/Qwen2.5-VL-3B-Instruct

Adapter

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Details

Model Description

Developed by: Rukiye Berna Turan
Model type: Vision-Language Model (VLM) LoRA Adapter
Language(s) (NLP): English
License: Apache-2.0
Finetuned from model: Qwen/Qwen2.5-VL-3B-Instruct

Model Sources

Repository: https://github.com/rukiyeberna/vlm-food-extractor
Base Model Card: https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct

Uses

Direct Use

Out-of-Scope Use

Medical allergen confirmation or clinical dietary advice.
Non-food visual document parsing.
Precise weight, volume, or caloric estimation.

Bias, Risks, and Limitations

The model may miss occluded, mixed, or interior ingredients that are not visually apparent on the plate surface.
Exact token matching can vary compared to colloquial culinary names (e.g., predicting tomato salad instead of caprese salad).

How to Get Started with the Model

python
import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import PeftModel

BASE_MODEL = "Qwen/Qwen2.5-VL-3B-Instruct"
ADAPTER_PATH = "rukiyeberna/qwen2.5-vl-3b-food-extract-lora"

# Load base model & LoRA adapter
base_model = AutoModelForImageTextToText.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
model = PeftModel.from_pretrained(base_model, ADAPTER_PATH)
processor = AutoProcessor.from_pretrained(ADAPTER_PATH)
model.eval()

# Load image and build prompt
image = Image.open("sample_food.jpg").convert("RGB")
prompt = """Analyze this image and return only valid JSON with exactly these keys:
- is_food
- image_title
- food_items
- drink_items"""

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": prompt},
        ],
    }
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], padding=True, return_tensors="pt").to(model.device)

with torch.inference_mode():
    generated_ids = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
        repetition_penalty=1.15,
        no_repeat_ngram_size=3
    )

output = processor.batch_decode(
    generated_ids[:, inputs.input_ids.shape[1]:], 
    skip_special_tokens=True
)[0]

print(output)

Training Details

Training Data

Trained on mrdbourke/FoodExtract-1k-Vision (1,510 training samples) preprocessed into LLaVA-style conversational format (image + structured JSON response).

Training Hyperparameters

Regime: LoRA PEFT (Vision encoder and LLM backbone frozen)[cite: 1]
Precision: bfloat16[cite: 1]
LoRA Rank ( $r$ ): 32[cite: 1]
LoRA Alpha ( $α$ ): 64
LoRA Dropout: 0.05[cite: 1]
Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj[cite: 1]

Evaluation

Testing Data & Metrics

Evaluated on unseen food test images across structured schema integrity, sequence similarity, and fuzzy set-based overlap[cite: 1]:

JSON Parsing Success Rate[cite: 1]
is_food Classification Accuracy[cite: 1]
Title Character Similarity (SequenceMatcher)[cite: 1]
Set-based Item $F 1 $ Score (Precision / Recall)[cite: 1]
ROUGE-1 / ROUGE-L Overlap

Technical Specifications

Architecture

Base Architecture: Qwen2.5-VL (3B Parameters)[cite: 1]
Adapter Type: Low-Rank Adaptation (LoRA) on Self-Attention and MLP Projections[cite: 1]
Hardware: Single GPU (NVIDIA T4 / A100 environment)

Framework Versions

PEFT: 0.19.1
Transformers: 4.49.0+
PyTorch: 2.x
qwen-vl-utils: Latest