YangC777

AGW-35B

README

License: apache-2.0

Overview

AGW-35B is a screenshot-based computer-use agent obtained by fine-tuning Qwen3.5-35B-A3B on 79,266 grounded and filtered GUI training examples generated by AutoGUIWorld. Given a task instruction, the current screenshot, and recent interaction history, the model predicts the next executable computer_use action.

AutoGUIWorld expands GUI training coverage without executing every trajectory in the corresponding software environment. A planner specifies atomic actions and their intended visual consequences; an image generator then synthesizes successive GUI states. The training set spans Chrome, Ubuntu, Windows, and macOS, including browser, office, creative, system, and scientific workflows.

AGW-35B is trained as a direct-action, non-thinking agent. Its response contains a short action description followed by one or more structured tool calls. Spatial actions use a resolution-independent 0–999 coordinate grid, which the executor maps to the current screen resolution.

Model at a Glance

Table with columns: Property, Value
Property	Value
Base model	Qwen3.5-35B-A3B
Architecture	Multimodal Mixture-of-Experts GUI agent
Primary input	Task instruction, screenshot, and recent interaction history
Primary output	`Action:` text and structured `computer_use` calls
Training examples	79,266 step-level GUI examples
Training domains	Chrome, Ubuntu, Windows, macOS
SFT context length	16,384 tokens
Weight format	BF16 Safetensors
Evaluation checkpoint	Step 469

Results

The final checkpoint is evaluated with the same task set as the base model. Interactive benchmarks allow up to 100 agent turns per task. OSWorld and Windows Agent Arena retain partial credit; macOSWorld and ScienceBoard report task success.

Table with columns: Benchmark, Evaluation set, Metric, Qwen3.5-35B-A3B, AGW-35B, Gain
Benchmark	Evaluation set	Metric	Qwen3.5-35B-A3B	AGW-35B	Gain
OSWorld	361 tasks	Mean task score	33.0	40.8	+7.8
Windows Agent Arena	154 tasks	Mean task score	19.4	27.9	+8.5
macOSWorld

AGW-35B also improves visual grounding on the 1,581-example English positive-target split of ScreenSpot-Pro.

Table with columns: ScreenSpot-Pro, Qwen3.5-35B-A3B, AGW-35B, Gain
ScreenSpot-Pro	Qwen3.5-35B-A3B	AGW-35B	Gain
Overall	31.7	57.1	+25.4
Icon targets	16.2	32.9	+16.7
Text targets	41.2	72.0	+30.7

All values are percentages. The metrics differ across benchmarks and should be interpreted within each benchmark rather than pooled across individual tasks.

Training Data

The AutoGUIWorld training set contains 79,266 examples distributed across four GUI environments:

Table with columns: Environment, Examples, Share
Environment	Examples	Share
Chrome	35,209	44.4%
Ubuntu	24,250	30.6%
Windows	14,778	18.6%
macOS	5,029	6.3%

Each example is derived from a multi-step screenshot trajectory and contains the task, current observation, recent action history, and the next action target. AutoGUIWorld provides spatial grounding for pointing actions and applies transition-level quality control before examples enter training.

Training Configuration

AGW-35B was trained with ms-swift Megatron on 32 GPUs across four nodes. The language model was fine-tuned while the visual encoder was frozen.

Table with columns: Setting, Value
Setting	Value
Epochs	3
Peak learning rate	1e-5
Minimum learning rate	1e-6
Warmup fraction	0.03
Weight decay	0.1
Global batch size	512
Micro-batch size	1
Maximum sequence length	16,384
Tensor / pipeline / data parallelism	2 / 1 / 16

Overlength examples were removed rather than truncated. The checkpoint uses the Qwen3.5 MoE architecture with 256 experts and 8 selected experts per token.

Interaction Contract

The model consumes screenshot-centered multi-turn messages and emits direct actions in the following form:

text

Action: Click the Settings icon.<tool_call>{"name":"computer_use","arguments":{"action":"left_click","coordinate":[927,54]}}</tool_call>

The action space covers mouse movement and clicks, dragging, keyboard input, typing, scrolling, waiting, and task termination. Coordinates are integers on a 0–999 grid. An external agent harness must execute the tool call, capture the next screenshot, and append the resulting turn to the interaction history.

AGW-35B was trained without target-side chain-of-thought. For consistent behavior, disable thinking at inference and preserve the repository's chat template.

Serving with vLLM

Use a recent Qwen3.5-compatible vLLM build. The following configuration matches the 16,384-token SFT context and the tensor-parallel setup used in evaluation:

bash

uv pip install vllm --torch-backend=auto \  --extra-index-url https://wheels.vllm.ai/nightly vllm serve YangC777/AGW-35B \  --tensor-parallel-size 2 \  --max-model-len 16384 \  --dtype bfloat16

A minimal single-step request through the OpenAI-compatible endpoint is shown below. In a full agent loop, replace the compact system message with the project harness prompt and append previous screenshot-action turns.

python

import base64from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") with open("screenshot.png", "rb") as f:    screenshot = base64.b64encode(f.read()).decode("utf-8") system_prompt = """You are a GUI agent. Given a task and screenshot, output exactly:Action: <one short imperative><tool_call>{"name":"computer_use","arguments":{"action":<action>, ...}}</tool_call>Use integer coordinates on a 0-999 grid. Use terminate when the task is complete.""" response = client.chat.completions.create(    model="YangC777/AGW-35B",    messages=[        {"role": "system", "content": system_prompt},        {            "role": "user",            "content": [                {                    "type": "image_url",                    "image_url": {"url": f"data:image/png;base64,{screenshot}"},                },                {                    "type": "text",                    "text": "# Task Instruction:\nOpen Settings and switch to dark mode.\n\n"                            "Generate the next move from the screenshot.",                },            ],        },    ],    temperature=0.0,    top_p=1.0,    max_tokens=1024,    extra_body={"chat_template_kwargs": {"enable_thinking": False}},) print(response.choices[0].message.content)

Intended Use

AGW-35B is intended for research and development of screenshot-based GUI agents, including:

cross-platform desktop and browser task execution;
action grounding in dense professional interfaces;
evaluation of synthetic GUI trajectories for agent post-training;
agent systems that execute model-produced actions inside controlled environments.

The checkpoint is specialized for computer use. It should be paired with an executor, environment reset mechanism, action parser, and benchmark-appropriate safety controls.

Limitations

The model observes rendered screenshots rather than hidden application or file-system state, so visually ambiguous states can lead to incorrect actions.
Evaluation uses English task instructions and English interfaces; performance in other languages has not been established.
Long workflows remain sensitive to early action errors and changes in application state.
Performance varies by application domain. In the reported evaluation, Chrome scores decrease on both OSWorld and Windows Agent Arena even as aggregate performance improves.
Training trajectories are generated through AutoGUIWorld and can inherit visual or transition artifacts from the generation pipeline.

Citation

AutoGUIWorld paper on arXiv

bibtex

@misc{yang2026autoguiworldimagegeneratorsvisual,      title={AutoGUIWorld: Image Generators as Visual World Models for GUI Agent},       author={Cheng Yang and Yifan Wu and Yutao Huang and Zhaohua Zhang and Beiduo Chen and Muxi Chen and Chenchen Zhao and Hexuan Deng and Haolin Yang and Geyuan Zhu and Sa Zhu and Jianhuan Zhuo and Qiuyong Xiao and Jianhao Ruan and Yiran Peng and Jiayi Zhang and Tian Ye and Xinlei Yu and Tianwen Jiang and Jihong Zhang and Yuyu Luo},      year={2026},      eprint={2610.01215},      archivePrefix={arXiv},      primaryClass={cs.CV},      url={https://arxiv.org/abs/2610.01215}, }

Acknowledgements

AGW-35B is built on Qwen3.5-35B-A3B. We thank the authors and maintainers of Qwen, OSWorld, Windows Agent Arena, macOSWorld, ScienceBoard, and ScreenSpot-Pro.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider