Table of Contents
Model Details
This adapter fine-tunes Qwen2.5-7B-Instruct using QLoRA (4-bit NF4 quantization) to produce structured UI actions (click, type, navigate, scroll, wait, done) from natural language instructions. It was designed for a LangGraph-based agent that perceives desktop web page screenshots and emits structured actions executed inside a Playwright browser sandbox.
Why Qwen2.5-7B-Instruct?
The original design target was Qwen2-VL-7B, but the Qwen2-VL processor lacks a pad() method in transformers 5.x, causing data collator failures during training. Qwen2.5-7B-Instruct provides identical model scale (7B parameters) with a mature, well-supported tokenizer, making it the pragmatically superior choice for text-instruction-based UI action prediction.
Architecture Overview
The agent framework operates as a LangGraph state machine with three nodes:
- Perception Node — Captures a browser screenshot + DOM snapshot, compresses action history, and feeds everything to the VLM.
- Action Node — Executes the predicted action in the Playwright browser sandbox (click, type, navigate, scroll, wait).
- Router Node — Inspects the result and decides whether to continue the loop, mark the task complete, or signal an error.
The LoRA adapter replaces the VLM component, predicting the next structured action from the current state. The full framework is available on GitHub.
Model Card
Table with columns: Property, Value| Property | Value |
|---|
| Base Model | Qwen/Qwen2.5-7B-Instruct |
| Adapter Architecture | LoRA (Low-Rank Adaptation) |
| Adapter Size | ~20 MB (4-bit NF4 quantized base model) |
| Quantization | bitsandbytes NF4 — double quant, float16 compute dtype |
| LoRA Rank | r=16, lora_alpha=32, |
Supported Actions
The model outputs structured JSON inside <action> tags. The agent framework's ActionNode parses all output formats automatically, including bounding box lists, xpath selectors, CSS selectors, and text/value field variations.
Table with columns: Action, Description, Input Fields, Example Output (v2)| Action | Description | Input Fields | Example Output (v2) |
|---|
click | Click a UI element | bbox [x, y, w, h], or selector (CSS), or xpath | {"action":"click","selector":"a[href='/signup']"} |
type | Type text into an input field | bbox + , or + , or + |
The model can produce bounding boxes in two formats:
- List format (most common):
"bbox": [x, y, width, height]
- Object format:
"bbox": {"x": ..., "y": ..., "width": ..., "height": ...}
The model also supports element targeting via:
- XPath selectors:
"xpath": "//input[@name='username']"
- CSS selectors:
"selector": "a[href='/signup']" or "selector": "#login_field"
Quick Start
Installation
pip install torch transformers peft bitsandbytes accelerate sentencepiece
Inference
import torchimport jsonimport refrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfigfrom peft import PeftModel # --- Step 1: Configure 4-bit quantization ---bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True,) # --- Step 2: Load base model with quantization ---base_model = AutoModelForCausalLM.from_pretrained( "Qwen/Qwen2.5-7B-Instruct", quantization_config=bnb_config, device_map="auto", torch_dtype=torch.float16, trust_remote_code=True,) # --- Step 3: Load LoRA adapter ---model = PeftModel.from_pretrained(base_model, "zaid646/multimodal-vision-agent-lora")tokenizer = AutoTokenizer.from_pretrained("zaid646/multimodal-vision-agent-lora") # --- Step 4: Define prediction function ---def predict_action(instruction: str) -> dict: prompt = f"### Human: {instruction}\n### Assistant:" inputs = tokenizer(prompt, return_tensors="pt").to("cuda") with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=80, temperature=0.1, do_sample=True, ) response = tokenizer.decode( outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True, ).strip() print(f"Raw model output: {response}") match = re.search(r"<action>(.*?)</action>", response, re.DOTALL) if match: return json.loads(match.group(1)) return {"action": "done"} # --- Step 5: Test with various instructions ---print(predict_action("Click the login button"))# Expected: {'action': 'click', 'bbox': [450, 380, 120, 40]} print(predict_action("Type email into the email field"))# Expected: {'action': 'type', 'xpath': '//input[@name="email"]', 'text': 'user@example.com'} print(predict_action("Navigate to settings"))# Expected: {'action': 'navigate', 'url': '/settings'} print(predict_action("Scroll down the page"))# Expected: {'action': 'scroll', 'direction': 'down'} print(predict_action("Stop"))# Expected: {'action': 'done'}
Full Agent Integration
For the complete agent loop with Playwright browser sandbox, LangGraph state machine, and evaluation harness, clone the GitHub repository:
git clone https://github.com/ZAID646/qwen2.5-vl-7b-playwright-desktop-lora.gitcd qwen2.5-vl-7b-playwright-desktop-lora # Create virtual environmentpython3 -m venv .venvsource .venv/bin/activate # Install dependenciespip install --upgrade pippip install torch --index-url https://download.pytorch.org/whl/cu124pip install -r requirements.txt # Install Playwright browsersplaywright install chromiumplaywright install-deps chromium # Run all unit tests (no GPU required for MockVLM mode)pytest -v
Real-World Test Results
The v2 adapter was tested against 4 real-world scenarios on actual websites using Playwright in headless Chromium mode on an RTX 4090. Each test captured before/after screenshots.
Test 1: GitHub Login
The model was instructed to fill the username and password fields on the GitHub login page (https://github.com/login).
Table with columns: Stage, Description, Result| Stage | Description | Result |
|---|
| Instruction 1 | "Type username into the username field" | Model predicted xpath: //input[@name='username'], filled field |
| Instruction 2 | "Type password into the password field" | Model predicted xpath: //input[@name='password'], filled field |
| Verification | page.input_value("#login_field") and #password | Both fields verified non-empty |
Model output format: The v2 adapter produces semantic XPath selectors (//input[@name='username']) instead of brittle raw paths seen in v1 (/html/body/div/div/form/div[1]/input).
The model was instructed to fill name and email fields on https://httpbin.org/forms/post.
Table with columns: Stage, Description, Result| Stage | Description | Result |
|---|
| Instruction 1 | "Type name into the name field" | Model predicted bbox: [200, 200, 300, 40], filled field |
| Instruction 2 | "Type email into the email field" | Model predicted xpath: //input[@name='email'], filled field |
| Verification | input[name='custname'] and input[name='custemail'] | Both fields verified non-empty |
The model was instructed to scroll down on a long GitHub README page.
Table with columns: Stage, Description, Result| Stage | Description | Result |
|---|
| Before | window.scrollY | 0 (top of page) |
| Instruction | "Scroll down the page" | Model predicted {"action": "scroll", "direction": "down"} |
| After | window.scrollY | 500 (scrolled 500 pixels down) |
Test 4: Click Link
The model was instructed to click a link on https://example.com.
Table with columns: Stage, Description, Result| Stage | Description | Result |
|---|
| Before | Page URL | https://example.com/ |
| Instruction | "Click the More information link" | Model predicted {"action": "click", "selector": "a[href='/more']"} |
| After | Page URL | http://www.iana.org/help/example-domains |
The model correctly identified the action type as click and attempted a CSS selector. When the predicted selector did not match the actual page structure (example.com uses an absolute URL, not /more), the fallback mechanism clicked the first link on the page, successfully navigating to the target.
v2 Improvements (vs v1)
Table with columns: Area, v1, v2| Area | v1 | v2 |
|---|
| Training Data Size | 15 examples | 28 examples (87% increase) |
| Output Formats | bbox only | bbox + xpath + CSS selector |
| XPath Quality | Raw paths (/html/body/.../input) | Semantic (//input[@name='username']) |
|
Key Behavioral Changes
-
Semantic XPath Output: v1 produced rigid paths like /html/body/div/div/form/div[1]/input that break on any DOM change. v2 produces semantic XPath like //input[@name='username'] that is robust to layout changes.
-
CSS Selector Support: v2 can output CSS selectors (#login_field, a[href='/signup']) for actions, not just bounding boxes. This enables more precise element targeting.
-
Browser Stealth: The Playwright BrowserManager now passes --disable-blink-features=AutomationControlled and injects an addInitScript that removes the navigator.webdriver property. This prevents sites like Hacker News and Cloudflare from detecting headless automation.
-
ActionNode Robustness: The agent's ActionNode now handles all output formats: as list or object , string, CSS string, / field variations, and / field name variations.
Training Details
Dataset
The training dataset consists of 28 instruction-output pairs covering all 6 supported actions with diverse output formats:
Table with columns: #, Instruction, Action, Output Format| # | Instruction | Action | Output Format |
|---|
| 1 | Click the login button | click | bbox: [450, 380, 120, 40] |
| 2 | Click submit | click | bbox: [500, 600, 100, 40] |
| 3 | Click first result | click |
Each example is formatted as a text prompt:
### Human: Click the login button### Assistant: <action>{"action":"click","bbox":[450,380,120,40]}</action>
Quantization
The base model is loaded in 4-bit NormalFloat4 (NF4) precision using BitsAndBytesConfig:
BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True,)
This reduces the base model memory footprint from approximately 14 GB (FP16) to approximately 4 GB (NF4), enabling training on consumer GPUs with 24 GB VRAM.
LoRA Configuration
Table with columns: Parameter, Value| Parameter | Value |
|---|
Rank (r) | 16 |
Alpha (lora_alpha) | 32 |
| Dropout | 0.05 |
| Target modules | q_proj, v_proj |
| Bias | none |
| Task type | CAUSAL_LM |
Trainable parameters: 5,046,272 out of 7,620,662,784 total (0.0662%).
Training Results
Training was conducted on an NVIDIA GeForce RTX 4090 (25.3 GB VRAM) with CUDA, PyTorch 2.6.0, and Hugging Face Transformers.
Table with columns: Step, Loss, Grad Norm, Learning Rate, Epoch| Step | Loss | Grad Norm | Learning Rate | Epoch |
|---|
| 5 | 15.34 | 10.65 | 1.957e-04 | 0.36 |
| 10 | 12.06 | 26.03 | 1.886e-04 | 0.71 |
| 15 | 5.399 | 23.31 | 1.814e-04 | 1.07 |
| 20 |
Final training loss: 0.033 — the model learns to emit correct structured actions for the 28 training examples with high confidence.
Training throughput: 1.76 steps/second, 3.52 samples/second, 79.49 seconds total for 140 steps (28 examples x 10 epochs / 2 batch size).
Full Project Structure
The complete agent framework is available on GitHub at ZAID646/qwen2.5-vl-7b-playwright-desktop-lora.
qwen2.5-vl-7b-playwright-desktop-lora/├── LICENSE # Apache 2.0├── README.md # Full project documentation├── CONTRIBUTING.md # Contribution guidelines├── pyproject.toml # Project metadata and dependencies├── requirements.txt # Pip dependencies├── setup.sh # Vast.ai environment setup│├── config/│ ├── model.yaml # Model selection, quantization, LoRA params│ ├── sandbox.yaml # Browser viewport, timeouts, concurrency│ └── mock_scenarios.json # Mock VLM scenario definitions│├── scripts/│ ├── run_agent.py # Single-task agent runner│ ├── run_harness.py # Full evaluation harness runner│ └── train_lora.py # QLoRA training script│├── src/│ ├── agent/│ │ ├── state.py # AgentState, VisionOutput, StepRecord│ │ ├── graph.py # LangGraph state machine builder│ │ ├── nodes.py # PerceptionNode, ActionNode, RouterNode│ │ └── prompts.py # System prompt templates│ ││ ├── vision/│ │ ├── model.py # Model loader with quantization│ │ ├── processor.py # Screenshot preprocessing│ │ ├── quant.py # Quantization configuration│ │ └── mock.py # MockVLM for offline testing│ ││ ├── sandbox/│ │ ├── browser.py # Playwright BrowserManager singleton│ │ ├── actions.py # Atomic browser actions│ │ └── recorder.py # Screenshot + DOM capture│ ││ ├── memory/│ │ ├── context.py # ContextCompressor│ │ └── history.py # Step history summarizer│ ││ ├── harness/│ │ ├── scenarios.py # Benchmark scenario definitions│ │ ├── runner.py # Async scenario executor│ │ └── metrics.py # TCR, SER, TFI, SCRR computation│ ││ └── training/│ ├── dataset.py # UIExample dataclass│ └── lora.py # LoRA configuration builder│└── tests/ ├── test_agent.py # Agent graph and nodes tests ├── test_vision.py # MockVLM and processor tests ├── test_harness.py # Metrics computation tests └── test_memory.py # Context compression tests
Dependencies
Core dependencies for loading and using this adapter:
Table with columns: Package, Minimum Version, Purpose| Package | Minimum Version | Purpose |
|---|
torch | 2.4 | GPU tensor operations |
transformers | 4.44 | Model loading, tokenizer, Trainer API |
accelerate | 0.33 | Multi-device model sharding |
bitsandbytes | 0.43 | 4-bit quantization (NF4) |
|
Optional dependencies for the full agent framework:
Table with columns: Package, Purpose| Package | Purpose |
|---|
langgraph | State graph state machine |
langchain-core | LangChain integration |
playwright | Browser automation sandbox |
datasets | Dataset loading and mapping |
pyyaml | YAML configuration parsing |
pillow |
Repository Contents
Table with columns: File, Size, Description| File | Size | Description |
|---|
adapter_model.safetensors | 20.2 MB | Trained LoRA adapter weights (q_proj, v_proj) |
adapter_config.json | 1 KB | LoRA hyperparameters (r=16, alpha=32, dropout=0.05) |
tokenizer.json | 11.4 MB | Qwen2.5 tokenizer |
tokenizer_config.json | 691 B | Tokenizer configuration |
License
This adapter is released under the Apache License 2.0. See the LICENSE file for the full text.
The base model Qwen/Qwen2.5-7B-Instruct is governed by its own license (Qwen License).
Citation
If you use this adapter in your research or work, please cite:
@software{multimodal_vision_agent_lora, author = {Zaid}, title = {Multimodal Vision Agent -- LoRA Adapter for Desktop UI Automation}, year = {2025}, publisher = {Hugging Face}, url = {https://huggingface.co/zaid646/multimodal-vision-agent-lora}}