MuhibBeekun

pickai-qwen2.5-vl-3b-layout-vision-lora

Evaluation (30-sample holdout)

Table with columns: Metric, Base Qwen2.5-VL-3B, This LoRA
Metric	Base Qwen2.5-VL-3B	This LoRA
Aggregate score	59.15%	93.24%
Location recall	54.72%	100.00%
Location coordinate match	0.00%	72.96%
JSON valid	100.00%	100.00%

Value gate: PASS (LoRA aggregate exceeds base on held-out synthetic floor plans).

Full eval log: fine-tune-eval-vision.md

What it does

mermaid
flowchart LR
  IMG[Warehouse floor plan PNG] --> VLM[Qwen2.5-VL-3B + LoRA]
  VLM --> JSON[facility_draft.locations JSON]
  JSON --> Profile[PickAI FacilityProfile]
  Profile --> Opt[Pick-path optimizer]

Each location object:

Table with columns: Field, Type, Example
Field	Type	Example
`location_id`	string	`A1-L01` (exact label on image)
`x`, `y`	number	Bay coordinates (meters)
`aisle`	string \| null	`A1`
`level`

Training mode: locations-only (shorter JSON targets; zones/docks inferred separately or merged later).

Training

Table with columns: Setting, Value
Setting	Value
Base model	Qwen/Qwen2.5-VL-3B-Instruct
Method	QLoRA (4-bit NF4) + PEFT LoRA
Rank (`r`)	32
LoRA alpha	64
LoRA dropout	0.05
Target modules	`q_proj`, `k_proj`, `v_proj`, , , ,

Synthetic data: generated locally via scripts/generate_layout_vision_jsonl.py (240 labeled floor-plan renders).

Training script: scripts/train_lora_layout_vision.py

Usage

PickAI runtime (recommended)

powershell
PICKAI_USE_VISION_LORA=1
PICKAI_VISION_TRAIN_MODE=locations_only
PICKAI_VISION_LORA_DIR=outputs/lora-vision

Docker Compose mounts this adapter read-only for the Facility Console layout upload flow.

Standalone (Transformers + PEFT)

python
import json
import torch
from peft import PeftModel
from transformers import AutoProcessor, BitsAndBytesConfig, Qwen2_5_VLForConditionalGeneration
from PIL import Image

base = "Qwen/Qwen2.5-VL-3B-Instruct"
adapter = "MuhibBeekun/pickai-qwen2.5-vl-3b-layout-vision-lora"

processor = AutoProcessor.from_pretrained(adapter, trust_remote_code=True)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    base,
    trust_remote_code=True,
    device_map="auto",
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.float16,
    ),
)
model = PeftModel.from_pretrained(model, adapter)
model.eval()

image = Image.open("floor_plan.png").convert("RGB")
prompt = (
    'Return JSON only: {"facility_draft":{"locations":[...]}} extracted from the warehouse floor plan image.\n'
    "Copy location_id exactly as printed on bay labels.\n"
)

messages = [{"role": "user", "content": [{"type": "image", "image": image}, {"type": "text", "text": prompt}]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)

with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=2048, do_sample=False)

decoded = processor.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(decoded)

Set PICKAI_VISION_TRAIN_MODE=locations_only when using PickAI inference helpers.

Limitations

Trained on synthetic top-down floor plans (2–3 aisles, 3–4 bays per side). Real CAD/PDF layouts may need fine-tuning or human review.
Locations-only output: aisles, docks, zones, and layout dimensions may need separate extraction or manual entry.
Requires GPU with ~8 GB+ VRAM for 4-bit inference; training used 24 GB.
Coordinate accuracy (~73% within tolerance on holdout) is good but not perfect — validate before production slotting.

Links

Table with columns: Resource, URL
Resource	URL
PickAI repository	https://github.com/Muhib-Beekun/pickai
Eval report	https://github.com/Muhib-Beekun/pickai/blob/main/docs/fine-tune-eval-vision.md
NL-parse dataset (separate task)	https://huggingface.co/datasets/MuhibBeekun/pickai-synthetic-nl-parse-v1

Citation

If you use this adapter in research or a product demo, please link to the PickAI repo and this model card.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

MuhibBeekun

Model Tree

Base

Qwen/Qwen2.5-VL-3B-Instruct

Adapter

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

Evaluation (30-sample holdout)

Table with columns: Metric, Base Qwen2.5-VL-3B, This LoRA
Metric	Base Qwen2.5-VL-3B	This LoRA
Aggregate score	59.15%	93.24%
Location recall	54.72%	100.00%
Location coordinate match	0.00%	72.96%
JSON valid	100.00%	100.00%

Value gate: PASS (LoRA aggregate exceeds base on held-out synthetic floor plans).

Full eval log: fine-tune-eval-vision.md

What it does

mermaid
flowchart LR
  IMG[Warehouse floor plan PNG] --> VLM[Qwen2.5-VL-3B + LoRA]
  VLM --> JSON[facility_draft.locations JSON]
  JSON --> Profile[PickAI FacilityProfile]
  Profile --> Opt[Pick-path optimizer]

Each location object:

Table with columns: Field, Type, Example
Field	Type	Example
`location_id`	string	`A1-L01` (exact label on image)
`x`, `y`	number	Bay coordinates (meters)
`aisle`	string \| null	`A1`
`level`

Training mode: locations-only (shorter JSON targets; zones/docks inferred separately or merged later).

Training

Table with columns: Setting, Value
Setting	Value
Base model	Qwen/Qwen2.5-VL-3B-Instruct
Method	QLoRA (4-bit NF4) + PEFT LoRA
Rank (`r`)	32
LoRA alpha	64
LoRA dropout	0.05
Target modules	`q_proj`, `k_proj`, `v_proj`, , , ,

Synthetic data: generated locally via scripts/generate_layout_vision_jsonl.py (240 labeled floor-plan renders).

Training script: scripts/train_lora_layout_vision.py

Usage

PickAI runtime (recommended)

powershell
PICKAI_USE_VISION_LORA=1
PICKAI_VISION_TRAIN_MODE=locations_only
PICKAI_VISION_LORA_DIR=outputs/lora-vision

Docker Compose mounts this adapter read-only for the Facility Console layout upload flow.

Standalone (Transformers + PEFT)

python
import json
import torch
from peft import PeftModel
from transformers import AutoProcessor, BitsAndBytesConfig, Qwen2_5_VLForConditionalGeneration
from PIL import Image

base = "Qwen/Qwen2.5-VL-3B-Instruct"
adapter = "MuhibBeekun/pickai-qwen2.5-vl-3b-layout-vision-lora"

processor = AutoProcessor.from_pretrained(adapter, trust_remote_code=True)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    base,
    trust_remote_code=True,
    device_map="auto",
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.float16,
    ),
)
model = PeftModel.from_pretrained(model, adapter)
model.eval()

image = Image.open("floor_plan.png").convert("RGB")
prompt = (
    'Return JSON only: {"facility_draft":{"locations":[...]}} extracted from the warehouse floor plan image.\n'
    "Copy location_id exactly as printed on bay labels.\n"
)

messages = [{"role": "user", "content": [{"type": "image", "image": image}, {"type": "text", "text": prompt}]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)

with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=2048, do_sample=False)

decoded = processor.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(decoded)

Set PICKAI_VISION_TRAIN_MODE=locations_only when using PickAI inference helpers.

Limitations

Trained on synthetic top-down floor plans (2–3 aisles, 3–4 bays per side). Real CAD/PDF layouts may need fine-tuning or human review.
Locations-only output: aisles, docks, zones, and layout dimensions may need separate extraction or manual entry.
Requires GPU with ~8 GB+ VRAM for 4-bit inference; training used 24 GB.
Coordinate accuracy (~73% within tolerance on holdout) is good but not perfect — validate before production slotting.

Links

Table with columns: Resource, URL
Resource	URL
PickAI repository	https://github.com/Muhib-Beekun/pickai
Eval report	https://github.com/Muhib-Beekun/pickai/blob/main/docs/fine-tune-eval-vision.md
NL-parse dataset (separate task)	https://huggingface.co/datasets/MuhibBeekun/pickai-synthetic-nl-parse-v1

Citation

If you use this adapter in research or a product demo, please link to the PickAI repo and this model card.

pickai-qwen2.5-vl-3b-layout-vision-lora

README

Evaluation (30-sample holdout)

What it does

Training

Usage

PickAI runtime (recommended)

Standalone (Transformers + PEFT)

Limitations

Links

Citation

Explore FriendliAI today

README

Evaluation (30-sample holdout)

What it does

Training

Usage

PickAI runtime (recommended)

Standalone (Transformers + PEFT)

Limitations

Links

Citation