Training
Fine-tuned from Qwen3-VL-8B-Instruct on ScreenRef.
Details will be described in the paper.
Results
Accuracy (%) is the fraction of predicted click points that land inside the ground-truth box.
Coordinates are mapped back to pixel space and checked against the box exactly, with no rounding
tolerance.
Table with columns: ScreenSpot-v2, ScreenSpot-Pro, MMBench-GUI L2, UI-Vision, OSWorld-G| ScreenSpot-v2 | ScreenSpot-Pro | MMBench-GUI L2 | UI-Vision | OSWorld-G |
|---|
| 95.2 (1211/1272) | 63.9 (1011/1581) | 86.8 (3121/3594) | 38.3 (2101/5479) | 65.4 (369/564) |
- Inference used vLLM with greedy decoding (temperature 0) and
max_tokens 1024.
- OSWorld-G is the original 564-question split.
Usage
The model expects a specific system prompt, and the chat template does not add it for you. Use the
evaluation code in GroundGUI (scripts/eval.sh) to
reproduce the numbers above; it sets the system prompt (scalecua_toolcall) and the scoring for
each benchmark.
The output looks like:
<tool_call>{"name": "computer_use", "arguments": {"action": "left_click", "coordinate": [x, y]}}</tool_call>
To get pixel coordinates from the output: x_px = x / 1000 * image_width and
y_px = y / 1000 * image_height.
Notes
- The tool name depends on the platform:
computer_use for desktop, browser_use for web,
mobile_use for mobile.
generation_config.json is inherited from the base model and enables sampling. Pass
do_sample=False to reproduce the numbers above.
- The weights were saved with transformers 5.12.
License
Apache License 2.0, the same as the base model. This model is a modified version of
Qwen3-VL-8B-Instruct: Prentis AI fine-tuned the
original weights.
The training data, ScreenRef, is distributed
under its own terms; this license covers the model only and grants no rights in the dataset or in
the screenshots it contains.
Citation
If you use this model, please cite the ScreenRef dataset:
PrentisAI/ScreenRef. A BibTeX entry will be
added when the paper is public.