OpenCUA-7B-converted API & Inference Endpoint

Installation & Download

First, install the required transformers dependencies:

bash
conda create -n opencua python=3.10
conda activate opencua
pip install -r requirement.txt

Download the model weight from huggingface:

bash
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id="xlangai/OpenCUA-7B",
    local_dir="OpenCUA-7B",                
    local_dir_use_symlinks=False  
)

🎯 GUI Grounding

The following code demonstrates how to use OpenCUA models for GUI grounding tasks:

python
import base64
import torch
from transformers import AutoTokenizer, AutoModel, AutoImageProcessor
from PIL import Image
import json

def encode_image(image_path: str) -> str:
    """Encode image to base64 string for model input."""
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode()

def load_opencua_model(model_path: str):
    """Load OpenCUA model, tokenizer, and image processor."""
    tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
    model = AutoModel.from_pretrained(
        model_path, 
        torch_dtype="auto", 
        device_map="auto", 
        trust_remote_code=True
    )
    image_processor = AutoImageProcessor.from_pretrained(model_path, trust_remote_code=True)
    
    return model, tokenizer, image_processor

def create_grounding_messages(image_path: str, instruction: str):
    """Create chat messages for GUI grounding task."""
    system_prompt = (
        "You are a GUI agent. You are given a task and a screenshot of the screen. "
        "You need to perform a series of pyautogui actions to complete the task."
    )
    
    messages = [
        {"role": "system", "content": system_prompt},
        {
            "role": "user",
            "content": [
                {"type": "image", "image": f"data:image/png;base64,{encode_image(image_path)}"},
                {"type": "text", "text": instruction},
            ],
        },
    ]
    return messages

def run_inference(model, tokenizer, image_processor, messages, image_path):
    """Run inference on the model."""
    # Prepare text input
    input_ids = tokenizer.apply_chat_template(
        messages, tokenize=True, add_generation_prompt=True
    )
    input_ids = torch.tensor([input_ids]).to(model.device)
    
    # Prepare image input  
    image = Image.open(image_path).convert('RGB')
    image_info = image_processor.preprocess(images=[image])
    pixel_values = torch.tensor(image_info['pixel_values']).to(
        dtype=torch.bfloat16, device=model.device
    )
    grid_thws = torch.tensor(image_info['image_grid_thw'])
    
    # Generate response
    with torch.no_grad():
        generated_ids = model.generate(
            input_ids,
            pixel_values=pixel_values,
            grid_thws=grid_thws,
            max_new_tokens=512,
            temperature=0
        )
    
    # Decode output
    prompt_len = input_ids.shape[1]
    generated_ids = generated_ids[:, prompt_len:]
    output_text = tokenizer.batch_decode(
        generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
    )[0]
    
    return output_text

# Example usage
model_path = "OpenCUA/OpenCUA-7B"  # or other model variants
image_path = "screenshot.png"
instruction = "Click on the submit button"

# Load model
model, tokenizer, image_processor = load_opencua_model(model_path)

# Create messages and run inference
messages = create_grounding_messages(image_path, instruction)
result = run_inference(model, tokenizer, image_processor, messages, image_path)

print("Model output:", result)

You can also run the five grounding examples in OpenCUA/model/inference/huggingface_inference.py:

markdown
cd ./model/inference/
python huggingface_inference.py

🖥️ Computer Use Agent

OpenCUAAgent is developed in the OSWorld environment based on OpenCUA models. It iteratively perceives the environment via screenshots, produces reflective long CoT as inner monologue, and predicts the next action to be executed. OpenCUAAgent uses 3 images in total and L2 CoT format in default.

Command for running OpenCUA-7B and OpenCUA-32B in OSWorld:

markdown
python run_multienv_opencua.py \
        --headless \
        --observation_type screenshot \
        --model OpenCUA-32B \
        --result_dir ./results --test_all_meta_path evaluation_examples/test_all_no_gdrive.json \
        --max_steps 100 \
        --num_envs 30  \
        --coordinate_type qwen25

AgentNet Dataset - Large-Scale Computer-Use Dataset

AgentNet is the first large-scale desktop computer-use agent trajectory dataset, containing 22.6K human-annotated computer-use tasks across Windows, macOS, and Ubuntu systems.

👉 AgentNet Huggingface Dataset

Download the dataset here：

markdown
pip install -U huggingface_hub
huggingface-cli download xlangai/AgentNet --repo-type dataset --local-dir ./AgentNet

Collecting computer-use agent training data requires 3 steps:

Demonstrate human computer-use task via AgentNetTool;
Preprocess the demonstration using Action Reduction & State-Action Matching;
For each step, synthesize reflective long CoT

1 AgentNetTool – Annotation & Verification Tool

Our AgentNetTool is a cross-platform GUI recorder that runs unobtrusively on annotators’ machines. It captures synchronized screen video, mouse/keyboard events, and accessibility trees, then provides an in-browser UI for reviewing, trimming, and submitting demonstrations. AgentNet Tool is available on Windows, macOS and Ubuntu.

👉 AgentNetTool Document

2 DataProcessor – Action Reduction & State–Action Matching

Raw demonstrations can contain thousands of low-level events that are too dense for model training.
The DataProcessor module (./data/data-process/) performs two key steps:

Action Reduction — merges granular signals into concise, semantically meaningful PyAutoGUI actions (e.g., collapsing mouse moves → click, coalescing scrolls, grouping key-press sequences into text or hotkeys).
State–Action Matching — aligns every reduced action with the last visually distinct frame before the action begins, avoiding future-information leakage and yielding compact state–action pairs.

These processed trajectories underlie all downstream training and evaluation.

3 CoTGenerator – Synthesizing Reflective Long Chain-of-Thought Inner Monologue

To boost robustness and interpretability, we augment each trajectory with reflective long Chain-of-Thought (CoT) reasoning.
The CoTGenerator pipeline (./data/cot-generator/) synthesizes step-level reflections that:

reflect on the previous action,
explain why an action is chosen given the current observation and history,
note potential alternative actions, and
forecast the expected next state.

Empirically, models trained with these rich CoTs scale better with data and generalize across unseen applications.

Evaluation

AgentNetBench (./AgentNetBench/) provides a realistic offline evaluator for OS agent trajectories. It compares model-predicted low-level actions (click, moveTo, write, press, scroll, terminate, etc.) against ground-truth human actions and reports detailed metrics.

👉 See AgentNetBench/README.md for usage instructions.

TODO

vLLM Support

We are actively working with the vLLM team to add support for OpenCUA models.

Workaround: For now, please use the standard transformers library as shown in the examples above. We will update this section once vLLM support becomes available.

Training Code

OpenCUA models are developed based on the training infrastructure of Kimi Team. We are developting the training pipeline based on the open-source infrastructure as well.

Acknowledge

License

This project is licensed under the MIT License - see the LICENSE file in the root folder for details.

Research Use and Disclaimer

OpenCUA models are intended for research and educational purposes only.

Prohibited Uses

The model may not be used for any purpose or activity that violates applicable laws or regulations in any jurisdiction
Use for illegal, unethical, or harmful activities is strictly prohibited

Disclaimer

The authors, contributors, and copyright holders are not responsible for any illegal, unethical, or harmful use of the Software, nor for any direct or indirect damages resulting from such use
Use of the "OpenCUA" name, logo, or trademarks does not imply any endorsement or affiliation unless separate written permission is obtained
Users are solely responsible for ensuring their use complies with applicable laws and regulations

Important Notes on Coordinate Systems

OpenCUA models use different coordinate systems depending on the base model:

OpenCUA-Qwen2-7B: Outputs relative coordinates (0.0 to 1.0 range)

python
# Example output: pyautogui.click(x=0.5, y=0.3)
# x=0.5 means 50% from left edge, y=0.3 means 30% from top edge

# Convert to absolute coordinates:
def qwen2_relative_to_absolute(rel_x, rel_y, original_width, original_height):
    abs_x = int(rel_x * original_width)
    abs_y = int(rel_y * original_height)
    return abs_x, abs_y

OpenCUA-7B and OpenCUA-32B (Qwen2.5-based): Output absolute coordinates after smart resize

python
# Example output: pyautogui.click(x=960, y=324)  
# These are coordinates on the smart-resized image, not the original image

# Convert to original image coordinates:
# Please refer to the smart_resize function in: https://github.com/huggingface/transformers/blob/67ddc82fbc7e52c6f42a395b4a6d278c55b77a39/src/transformers/models/qwen2_vl/image_processing_qwen2_vl.py#L55
def qwen25_smart_resize_to_absolute(model_x, model_y, original_width, original_height):
    # First, calculate the smart-resized dimensions
    resized_height, resized_width = smart_resize(original_height, original_width, factor = 28, min_pixels = 3136, max_pixels = 12845056)
    
    # Convert model output to relative coordinates on original image
    rel_x = model_x / resized_width
    rel_y = model_y / resized_height
    
    # Then convert to absolute coordinates on original image
    abs_x = int(rel_x * original_width)
    abs_y = int(rel_y * original_height)
    return abs_x, abs_y

Citation

If you use OpenCUA models in your research, please cite our work:

bibtex
@misc{wang2025opencuaopenfoundationscomputeruse,
      title={OpenCUA: Open Foundations for Computer-Use Agents}, 
      author={Xinyuan Wang and Bowen Wang and Dunjie Lu and Junlin Yang and Tianbao Xie and Junli Wang and Jiaqi Deng and Xiaole Guo and Yiheng Xu and Chen Henry Wu and Zhennan Shen and Zhuokai Li and Ryan Li and Xiaochuan Li and Junda Chen and Boyuan Zheng and Peihang Li and Fangyu Lei and Ruisheng Cao and Yeqiao Fu and Dongchan Shin and Martin Shin and Jiarui Hu and Yuyan Wang and Jixuan Chen and Yuxiao Ye and Danyang Zhang and Dikang Du and Hao Hu and Huarong Chen and Zaida Zhou and Haotian Yao and Ziwei Chen and Qizheng Gu and Yipu Wang and Heng Wang and Diyi Yang and Victor Zhong and Flood Sung and Y. Charles and Zhilin Yang and Tao Yu},
      year={2025},
      eprint={2508.09123},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2508.09123}, 
}

Installation & Download

First, install the required transformers dependencies:

bash
conda create -n opencua python=3.10
conda activate opencua
pip install -r requirement.txt

Download the model weight from huggingface:

bash
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id="xlangai/OpenCUA-7B",
    local_dir="OpenCUA-7B",                
    local_dir_use_symlinks=False  
)

🎯 GUI Grounding

The following code demonstrates how to use OpenCUA models for GUI grounding tasks:

python
import base64
import torch
from transformers import AutoTokenizer, AutoModel, AutoImageProcessor
from PIL import Image
import json

def encode_image(image_path: str) -> str:
    """Encode image to base64 string for model input."""
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode()

def load_opencua_model(model_path: str):
    """Load OpenCUA model, tokenizer, and image processor."""
    tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
    model = AutoModel.from_pretrained(
        model_path, 
        torch_dtype="auto", 
        device_map="auto", 
        trust_remote_code=True
    )
    image_processor = AutoImageProcessor.from_pretrained(model_path, trust_remote_code=True)
    
    return model, tokenizer, image_processor

def create_grounding_messages(image_path: str, instruction: str):
    """Create chat messages for GUI grounding task."""
    system_prompt = (
        "You are a GUI agent. You are given a task and a screenshot of the screen. "
        "You need to perform a series of pyautogui actions to complete the task."
    )
    
    messages = [
        {"role": "system", "content": system_prompt},
        {
            "role": "user",
            "content": [
                {"type": "image", "image": f"data:image/png;base64,{encode_image(image_path)}"},
                {"type": "text", "text": instruction},
            ],
        },
    ]
    return messages

def run_inference(model, tokenizer, image_processor, messages, image_path):
    """Run inference on the model."""
    # Prepare text input
    input_ids = tokenizer.apply_chat_template(
        messages, tokenize=True, add_generation_prompt=True
    )
    input_ids = torch.tensor([input_ids]).to(model.device)
    
    # Prepare image input  
    image = Image.open(image_path).convert('RGB')
    image_info = image_processor.preprocess(images=[image])
    pixel_values = torch.tensor(image_info['pixel_values']).to(
        dtype=torch.bfloat16, device=model.device
    )
    grid_thws = torch.tensor(image_info['image_grid_thw'])
    
    # Generate response
    with torch.no_grad():
        generated_ids = model.generate(
            input_ids,
            pixel_values=pixel_values,
            grid_thws=grid_thws,
            max_new_tokens=512,
            temperature=0
        )
    
    # Decode output
    prompt_len = input_ids.shape[1]
    generated_ids = generated_ids[:, prompt_len:]
    output_text = tokenizer.batch_decode(
        generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
    )[0]
    
    return output_text

# Example usage
model_path = "OpenCUA/OpenCUA-7B"  # or other model variants
image_path = "screenshot.png"
instruction = "Click on the submit button"

# Load model
model, tokenizer, image_processor = load_opencua_model(model_path)

# Create messages and run inference
messages = create_grounding_messages(image_path, instruction)
result = run_inference(model, tokenizer, image_processor, messages, image_path)

print("Model output:", result)

You can also run the five grounding examples in OpenCUA/model/inference/huggingface_inference.py:

markdown
cd ./model/inference/
python huggingface_inference.py

🖥️ Computer Use Agent

Command for running OpenCUA-7B and OpenCUA-32B in OSWorld:

markdown
python run_multienv_opencua.py \
        --headless \
        --observation_type screenshot \
        --model OpenCUA-32B \
        --result_dir ./results --test_all_meta_path evaluation_examples/test_all_no_gdrive.json \
        --max_steps 100 \
        --num_envs 30  \
        --coordinate_type qwen25

AgentNet Dataset - Large-Scale Computer-Use Dataset

AgentNet is the first large-scale desktop computer-use agent trajectory dataset, containing 22.6K human-annotated computer-use tasks across Windows, macOS, and Ubuntu systems.

👉 AgentNet Huggingface Dataset

Download the dataset here：

markdown
pip install -U huggingface_hub
huggingface-cli download xlangai/AgentNet --repo-type dataset --local-dir ./AgentNet

Collecting computer-use agent training data requires 3 steps:

Demonstrate human computer-use task via AgentNetTool;
Preprocess the demonstration using Action Reduction & State-Action Matching;
For each step, synthesize reflective long CoT

1 AgentNetTool – Annotation & Verification Tool

👉 AgentNetTool Document

2 DataProcessor – Action Reduction & State–Action Matching

Raw demonstrations can contain thousands of low-level events that are too dense for model training.
The DataProcessor module (./data/data-process/) performs two key steps:

Action Reduction — merges granular signals into concise, semantically meaningful PyAutoGUI actions (e.g., collapsing mouse moves → click, coalescing scrolls, grouping key-press sequences into text or hotkeys).
State–Action Matching — aligns every reduced action with the last visually distinct frame before the action begins, avoiding future-information leakage and yielding compact state–action pairs.

These processed trajectories underlie all downstream training and evaluation.

3 CoTGenerator – Synthesizing Reflective Long Chain-of-Thought Inner Monologue

reflect on the previous action,
explain why an action is chosen given the current observation and history,
note potential alternative actions, and
forecast the expected next state.

Empirically, models trained with these rich CoTs scale better with data and generalize across unseen applications.

Evaluation

👉 See AgentNetBench/README.md for usage instructions.

TODO

vLLM Support

We are actively working with the vLLM team to add support for OpenCUA models.

Workaround: For now, please use the standard transformers library as shown in the examples above. We will update this section once vLLM support becomes available.

Training Code

OpenCUA models are developed based on the training infrastructure of Kimi Team. We are developting the training pipeline based on the open-source infrastructure as well.

Acknowledge

License

This project is licensed under the MIT License - see the LICENSE file in the root folder for details.

Research Use and Disclaimer

OpenCUA models are intended for research and educational purposes only.

Prohibited Uses

The model may not be used for any purpose or activity that violates applicable laws or regulations in any jurisdiction
Use for illegal, unethical, or harmful activities is strictly prohibited

Disclaimer

The authors, contributors, and copyright holders are not responsible for any illegal, unethical, or harmful use of the Software, nor for any direct or indirect damages resulting from such use
Use of the "OpenCUA" name, logo, or trademarks does not imply any endorsement or affiliation unless separate written permission is obtained
Users are solely responsible for ensuring their use complies with applicable laws and regulations

Important Notes on Coordinate Systems

OpenCUA models use different coordinate systems depending on the base model:

OpenCUA-Qwen2-7B: Outputs relative coordinates (0.0 to 1.0 range)

python
# Example output: pyautogui.click(x=0.5, y=0.3)
# x=0.5 means 50% from left edge, y=0.3 means 30% from top edge

# Convert to absolute coordinates:
def qwen2_relative_to_absolute(rel_x, rel_y, original_width, original_height):
    abs_x = int(rel_x * original_width)
    abs_y = int(rel_y * original_height)
    return abs_x, abs_y

OpenCUA-7B and OpenCUA-32B (Qwen2.5-based): Output absolute coordinates after smart resize

python
# Example output: pyautogui.click(x=960, y=324)  
# These are coordinates on the smart-resized image, not the original image

# Convert to original image coordinates:
# Please refer to the smart_resize function in: https://github.com/huggingface/transformers/blob/67ddc82fbc7e52c6f42a395b4a6d278c55b77a39/src/transformers/models/qwen2_vl/image_processing_qwen2_vl.py#L55
def qwen25_smart_resize_to_absolute(model_x, model_y, original_width, original_height):
    # First, calculate the smart-resized dimensions
    resized_height, resized_width = smart_resize(original_height, original_width, factor = 28, min_pixels = 3136, max_pixels = 12845056)
    
    # Convert model output to relative coordinates on original image
    rel_x = model_x / resized_width
    rel_y = model_y / resized_height
    
    # Then convert to absolute coordinates on original image
    abs_x = int(rel_x * original_width)
    abs_y = int(rel_y * original_height)
    return abs_x, abs_y

Citation

If you use OpenCUA models in your research, please cite our work:

bibtex
@misc{wang2025opencuaopenfoundationscomputeruse,
      title={OpenCUA: Open Foundations for Computer-Use Agents}, 
      author={Xinyuan Wang and Bowen Wang and Dunjie Lu and Junlin Yang and Tianbao Xie and Junli Wang and Jiaqi Deng and Xiaole Guo and Yiheng Xu and Chen Henry Wu and Zhennan Shen and Zhuokai Li and Ryan Li and Xiaochuan Li and Junda Chen and Boyuan Zheng and Peihang Li and Fangyu Lei and Ruisheng Cao and Yeqiao Fu and Dongchan Shin and Martin Shin and Jiarui Hu and Yuyan Wang and Jixuan Chen and Yuxiao Ye and Danyang Zhang and Dikang Du and Hao Hu and Huarong Chen and Zaida Zhou and Haotian Yao and Ziwei Chen and Qizheng Gu and Yipu Wang and Heng Wang and Diyi Yang and Victor Zhong and Flood Sung and Y. Charles and Zhilin Yang and Tao Yu},
      year={2025},
      eprint={2508.09123},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2508.09123}, 
}

OpenCUA-7B-converted

README

Installation & Download

🎯 GUI Grounding

🖥️ Computer Use Agent

AgentNet Dataset - Large-Scale Computer-Use Dataset

1 AgentNetTool – Annotation & Verification Tool

2 DataProcessor – Action Reduction & State–Action Matching

3 CoTGenerator – Synthesizing Reflective Long Chain-of-Thought Inner Monologue

Evaluation

TODO

vLLM Support

Training Code

Acknowledge

License

Research Use and Disclaimer

Prohibited Uses

Disclaimer

Important Notes on Coordinate Systems

Citation

Explore FriendliAI today

README

Installation & Download

🎯 GUI Grounding

🖥️ Computer Use Agent

AgentNet Dataset - Large-Scale Computer-Use Dataset

1 AgentNetTool – Annotation & Verification Tool

2 DataProcessor – Action Reduction & State–Action Matching

3 CoTGenerator – Synthesizing Reflective Long Chain-of-Thought Inner Monologue

Evaluation

TODO

vLLM Support

Training Code

Acknowledge

License

Research Use and Disclaimer

Prohibited Uses

Disclaimer

Important Notes on Coordinate Systems

Citation