Highlights
- Strict Raw Pass@1 Evaluation: Single greedy prediction (
do_sample=False), no image cropping, no iterative zoom, no planner, no pass@k, no reranking.
- Extreme Data & Compute Efficiency: Trained on ~7K curated samples (~60% click, ~40% refusal) on a ~$8 cloud GPU budget (initial QLoRA 4-bit adapter followed by 250 steps in fp16).
- Zero-Trust Verifiability: Unlike closed evaluations, this repository ships the complete per-sample output logs (
osworld_g_predictions.jsonl — 1,128 rows; predictions.jsonl — 5,479 rows) containing the exact prompt, raw model text, normalized and pixel coordinates, ground-truth bounding boxes/polygons, and SHA-256 checksums so anyone can recompute the scores offline in 2 seconds.
Benchmark Results & Reference Context
All AA-VL-4B scores below are computed from the shipped fp16 merged weights (model.safetensors) in a single greedy pass:
Table with columns: Benchmark, AA-VL-4B (Ours, 4B), Correct / Total, Reference Context (Published Numbers)*| Benchmark | AA-VL-4B (Ours, 4B) | Correct / Total | Reference Context (Published Numbers)* |
|---|
| OSWorld-G (Original) | 65.25% | 368 / 564 | Official Leaderboard: GUI-Owl-7B (55.9), UI-Venus-1.0-7B (54.6), Jedi-7B (54.1 on 4M samples), UI-TARS-1.5-7B (52.8), Qwen2.5-VL-7B (31.4). Recent 8B tech reports: MAI-UI-8B (60.1), UI-Venus-1.5-8B (69.7). |
| OSWorld-G (Refined) | 74.47% | 420 / 564 | Published 7B–8B reports: UI-Venus-1.0-7B (61.7), UI-Venus-1.5-8B (74.1). |
| UI-Vision (Final Pooled) | 29.93% | 1640 / 5479 | Official Leaderboard: UI-TARS-72B (25.50). Self-reported ≤8B: UI-Venus-1.0-7B (26.5), Phi-Ground (27.2), MAI-UI-2B (30.3), MAI-UI-8B (40.7), UI-Venus-1.5-8B (46.5). |
- UI-Vision Subset Breakdown: Basic 39.28% (
696/1772) · Functional 37.75% (669/1772) · Spatial 14.21% (275/1935).
- *Note on Reference Context: Published numbers for third-party models come from their respective papers/leaderboards and may differ in system prompt, resolution scaling, or evaluation harness. We include them solely to provide scale context for what a 4B model trained on ~7K samples achieves. Our own runs are 100% raw greedy pass@1 and fully auditable via the included
.jsonl files.
Instant Verification (No GPU Required)
Every raw generation, coordinate, and ground-truth box ships directly in this repo:
Table with columns: Benchmark, Artifact File, Rows, Score| Benchmark | Artifact File | Rows | Score |
|---|
| OSWorld-G (Original / Refined) | osworld_g_predictions.jsonl | 1,128 (564 × 2) | 368/564 (65.25%) / 420/564 (74.47%) |
| UI-Vision Final | predictions.jsonl | 5,479 | 1640/5479 (29.93%) |
Re-score both benchmarks from scratch in one line:
python3 -c "import json;r=[json.loads(l) for l in open('osworld_g_predictions.jsonl')];print({v:sum(a['correct'] for a in r if a['variant']==v) for v in ('original','refined')})"python3 -c "import json;r=[json.loads(l) for l in open('predictions.jsonl')];print(sum(x['correct'] for x in r),'/',len(r))"
See VERIFY.md and osworld_g_eval.py for full protocol details, coordinate mapping (pred_xy_px = pred_xy_norm999 / 1000 * screen_size), and SHA-256 hashes. You can also browse predictions visually in the Interactive OSWorld-G Demo Space.
from transformers import AutoProcessor, AutoModelForImageTextToTextfrom PIL import Image MODEL = "emirozcan16/AA-VL-4B"proc = AutoProcessor.from_pretrained(MODEL, trust_remote_code=True)model = AutoModelForImageTextToText.from_pretrained( MODEL, trust_remote_code=True, torch_dtype="auto", device_map="auto") img = Image.open("screen.png").convert("RGB")msgs = [{"role": "user", "content": [ {"type": "image", "image": img}, {"type": "text", "text": "Instruction: Click the funnel icon right of the search bar. " "Reply with the single action line now, nothing else:"}]}]txt = proc.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)inp = proc(text=[txt], images=[img], return_tensors="pt").to(model.device)out = model.generate(**inp, max_new_tokens=64, do_sample=False)print(proc.batch_decode(out[:, inp["input_ids"].shape[1]:], skip_special_tokens=True)[0])# -> click x=<int> y=<int> (0-999 normalized scale) or refuse
For the exact system prompt used in the benchmark runs, see SYS_CLICK in osworld_g_eval.py.
GGUF (llama.cpp / LM Studio) & LoRA Adapter
Training Data
~7K curated rows (~60% click, ~40% refusal). Sources:
Table with columns: Source, What we used, Role, License| Source | What we used | Role | License |
|---|
Jedi (xlangai/Jedi, HF) | icon/component grounding clicks (train_raw/val_raw, os_layout, component libraries) | click labels | see upstream |
| GroundCUA | GUI grounding click slice (~2.5K rows, picked_json + images) | click labels | MIT |
ScreenSpot-v2 (OS-Copilot/ScreenSpot-v2, HF) | desktop screenshots only | build synthetic refusal pairs | see upstream |
| Our synthesis | instruction ↔️ unrelated screenshot cross-pairing (doc–doc, icon–icon, desktop) | refusal examples | ours |
Base model: Qwen/Qwen3-VL-4B-Instruct (Apache-2.0). No private or scraped user data. The assembled training set is not released for now. We did not train on any benchmark's answer labels.
Eval Protocol & Limitations
- OSWorld-G: official point-in-box / point-in-polygon (510 click + 54 refusal = 564 samples), evaluated on both Original (
OSWorld-G.json) and Refined (OSWorld-G_refined.json) instructions.
- UI-Vision: official harness over its three subsets (5,479 samples total).
- Limitations: Coordinates are emitted on a
0–999 normalized grid. Single seed; benchmarks have not yet been independently reproduced by third parties (which is why all raw per-sample predictions are shipped in the repo).
Project
Built in ~3 months on a ~$8 compute budget. Contact: ozcannemir08@gmail.com