labos1

labos-vlm-7b

Deploy Dedicated

README

License: apache-2.0

Adapter Details

Base model: Qwen/Qwen2.5-VL-7B-Instruct
Adapter type: LoRA via PEFT / MS-SWIFT
Rank: 32
LoRA alpha: 64
LoRA dropout: 0.05
Target modules: Qwen language-model projection layers matching q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj
Training checkpoint: checkpoint-750
Training data: lab_supervision_25k_train.jsonl / lab_supervision_25k_val.jsonl

Training Parameters

Trainer: MS-SWIFT SFT with PEFT LoRA
Epochs: 2.0
Devices: 8 GPUs (global_world_size=8)
Per-device train batch size: 1
Gradient accumulation steps: 4
Effective global train batch size: 32 examples per optimizer step
Per-device eval batch size: 1
Learning rate: 1e-4
Scheduler: cosine
Warmup ratio:

Run With MS-SWIFT

Install the Qwen video runtime dependencies and MS-SWIFT in your environment, then run:

bash

swift infer \  --model Qwen/Qwen2.5-VL-7B-Instruct \  --adapters labos1/labos-vlm-7b \  --infer_backend pt

For local staged weights before upload:

bash

swift infer \  --model Qwen/Qwen2.5-VL-7B-Instruct \  --adapters /data/david/models/release/labos-vlm-7b \  --infer_backend pt

Run With Transformers And PEFT

python

from peft import PeftModelfrom transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor base_model = "Qwen/Qwen2.5-VL-7B-Instruct"adapter = "labos1/labos-vlm-7b" processor = AutoProcessor.from_pretrained(base_model)model = Qwen2_5_VLForConditionalGeneration.from_pretrained(    base_model,    torch_dtype="auto",    device_map="auto",)model = PeftModel.from_pretrained(model, adapter)

Example Monitoring Prompt

This adapter was fine-tuned with prompts similar to:

text

You are a real-time lab assistant monitoring a scientist's wet-lab procedure from short video windows.The current protocol state/history is provided below. Watch the current window and update the state.Report protocol errors only when supported by the visible time window or state.Ignore irrelevant unknown keys in the state JSON.Compare the protocol order, prior history, and watched window.Identify the main protocol step being performed in this watched video window.STATE:{"equipment":["P20 pipette","sterile tips","microcentrifuge tube rack"],"history":[{"step":"1","tas":20,"tds":20}],"objects":{},"on":"2","protocol":[{"desc":"Place the tubes in a clean rack.","order":1,"step":"1"},{"desc":"Attach a new sterile pipette tip.","order":2,"step":"2"},{"desc":"Aspirate reagent without touching the tube wall.","order":3,"step":"3"}],"protocol_summary":"Demonstration of aseptic pipetting technique.","reagents":["sample buffer"]}Return strict JSON only.## Response Format{  "explanation": "string",  "observed_step_id": "string or null"}## QuestionWhich protocol step is being performed in this video window?

Run python generate_monitoring_prompts.py from this folder to print example monitoring prompts that match the training/evaluation prompt style.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

labos1

Model Tree

Base

Qwen/Qwen2.5-VL-7B-Instruct

Adapter

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Adapter Details

Base model: Qwen/Qwen2.5-VL-7B-Instruct
Adapter type: LoRA via PEFT / MS-SWIFT
Rank: 32
LoRA alpha: 64
LoRA dropout: 0.05
Target modules: Qwen language-model projection layers matching q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj
Training checkpoint: checkpoint-750
Training data: lab_supervision_25k_train.jsonl / lab_supervision_25k_val.jsonl

Training Parameters

Trainer: MS-SWIFT SFT with PEFT LoRA
Epochs: 2.0
Devices: 8 GPUs (global_world_size=8)
Per-device train batch size: 1
Gradient accumulation steps: 4
Effective global train batch size: 32 examples per optimizer step
Per-device eval batch size: 1
Learning rate: 1e-4
Scheduler: cosine
Warmup ratio:

Run With MS-SWIFT

Install the Qwen video runtime dependencies and MS-SWIFT in your environment, then run:

bash

swift infer \  --model Qwen/Qwen2.5-VL-7B-Instruct \  --adapters labos1/labos-vlm-7b \  --infer_backend pt

For local staged weights before upload:

bash

swift infer \  --model Qwen/Qwen2.5-VL-7B-Instruct \  --adapters /data/david/models/release/labos-vlm-7b \  --infer_backend pt

Run With Transformers And PEFT

python

from peft import PeftModelfrom transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor base_model = "Qwen/Qwen2.5-VL-7B-Instruct"adapter = "labos1/labos-vlm-7b" processor = AutoProcessor.from_pretrained(base_model)model = Qwen2_5_VLForConditionalGeneration.from_pretrained(    base_model,    torch_dtype="auto",    device_map="auto",)model = PeftModel.from_pretrained(model, adapter)

Example Monitoring Prompt

This adapter was fine-tuned with prompts similar to:

text

You are a real-time lab assistant monitoring a scientist's wet-lab procedure from short video windows.The current protocol state/history is provided below. Watch the current window and update the state.Report protocol errors only when supported by the visible time window or state.Ignore irrelevant unknown keys in the state JSON.Compare the protocol order, prior history, and watched window.Identify the main protocol step being performed in this watched video window.STATE:{"equipment":["P20 pipette","sterile tips","microcentrifuge tube rack"],"history":[{"step":"1","tas":20,"tds":20}],"objects":{},"on":"2","protocol":[{"desc":"Place the tubes in a clean rack.","order":1,"step":"1"},{"desc":"Attach a new sterile pipette tip.","order":2,"step":"2"},{"desc":"Aspirate reagent without touching the tube wall.","order":3,"step":"3"}],"protocol_summary":"Demonstration of aseptic pipetting technique.","reagents":["sample buffer"]}Return strict JSON only.## Response Format{  "explanation": "string",  "observed_step_id": "string or null"}## QuestionWhich protocol step is being performed in this video window?

Run python generate_monitoring_prompts.py from this folder to print example monitoring prompts that match the training/evaluation prompt style.