Overview
AgentEye Grounder 0.8B v1 is an experimental fine-tuned
vision-language model designed to identify the screen region associated
with a user interface instruction.
Given a screenshot and an instruction such as:
"Tap the Work favorite."
the model is trained to return a JSON response containing a bounding box
and a click point:
{ "bbox": [9, 222, 192, 259], "point": [101, 240]}
The numbers above illustrate the expected output format; they are not
a guaranteed prediction for every screenshot.
This model is intended as a potential visual-grounding component for
desktop agents, including the Agent's EYE / Desk Agent direction. It
predicts locations; it does not itself operate the mouse or keyboard.
What it does
- Instruction-conditioned grounding: use a natural-language
instruction to identify a UI target in a screenshot.
- Bounding-box prediction: return
[x1, y1, x2, y2] for the
predicted target region.
- Click-point prediction: return
[cx, cy] as a candidate point
inside or near the target.
- Compact model scale: fine-tunes a small vision-language base
model rather than training a large model from scratch.
- Agent integration potential: output can be consumed by a
separate executor that validates coordinates before acting.
Model details
Field Details
Suggested model name AgentEye-Grounder-0.8B-v1
Model type Vision-language model (VLM)
Base model unsloth/Qwen3.5-0.8B (confirm this matches the exact checkpoint used for training)
Primary task Screenshot UI grounding
Input Screenshot + natural-language UI instruction
Output JSON with bbox and point
Training dataset Salesforce/grounding_dataset
Training stage Version 1, initial fine-tune
Training framework Unsloth + Hugging Face ecosystem
Intended use Research and experimentation with desktop agents
Before publishing: replace the Hugging Face links/placeholders
below with the actual model repository URL and verify that the
base-model name matches the checkpoint you trained.
Training summary
The provided training log reports the following configuration:
Setting Value
Dataset examples reported 67,356
Epochs 1
Total steps shown in the supplied log 3,000
Per-device batch size 2
Gradient accumulation steps 4
Data-parallel GPUs 1
Effective batch size 8
Trainable parameters 13,181,952
Total parameters 866,167,872
Trainable parameter share 1.52%
Step-count note: the project description says this is a
30,000-step version, but the supplied Unsloth log says
Total steps = 3,000. These values conflict. Confirm the actual
completed step count from the training run before publishing the model
card. Do not label the checkpoint as trained for 30,000 steps unless the
run really completed that many steps.
The trainable parameter count indicates that only a small portion of the
full model was updated, consistent with parameter-efficient fine-tuning.
The exact adapter configuration (for example, LoRA rank, target modules,
learning rate, optimizer, scheduler, sequence length, image resolution,
random seed, and hardware type) was not included in the supplied details
and should be added when confirmed.
Dataset
Training dataset:
Salesforce/grounding_dataset
The training data was described as containing 67,356 examples. The
model's target format is a JSON object with:
bbox: [x1, y1, x2, y2]
point: [cx, cy]
Dataset-specific details such as the exact split, filtering rules,
coordinate convention, license, and any preprocessing or augmentation
should be checked against the dataset card and the actual preprocessing
code before making further claims.
Please review and comply with the dataset's license and terms before
redistributing model weights or using the model commercially.
Quick start
The examples below assume that you have published a compatible merged
model to Hugging Face. Replace
YOUR_HF_USERNAME/AgentEye-Grounder-0.8B-v1 with your actual repository
ID.
1. Install dependencies
Run in a Python environment with a supported CUDA GPU:
pip install -U unsloth transformers pillow accelerate
For a fresh environment, follow the current installation instructions
for Unsloth and make sure your PyTorch/CUDA
versions are compatible. GPU inference is recommended; CPU inference may
be significantly slower.
2. Load the model
import torchfrom unsloth import FastVisionModel MODEL_ID = "Navanjana/AgentEye-Grounder-0.8B-v1-GGUF" model, tokenizer = FastVisionModel.from_pretrained( model_name=MODEL_ID, load_in_4bit=False, # use True only if the published model/Unsloth setup supports it auto_model=True,) FastVisionModel.for_inference(model)
If you publish an adapter-only checkpoint rather than a merged model,
load the exact compatible base model first and then load the adapter
according to the Unsloth/PEFT instructions. An adapter-only repository
is not interchangeable with a merged model repository.
3. Run a prediction on a screenshot
import jsonimport torchfrom PIL import Image # Replace with the path to your screenshot.image = Image.open("screenshot.png").convert("RGB")instruction = "Click the search icon." messages = [ { "role": "system", "content": ( "You are a screen executor. Given a screenshot and a clear " "instruction, locate the target UI element and return its position. " 'Always reply in this exact JSON format: ' '{"bbox": [x1, y1, x2, y2], "point": [cx, cy]}' ), }, { "role": "user", "content": [ {"type": "image"}, {"type": "text", "text": instruction}, ], },] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True,) inputs = tokenizer( image, prompt, add_special_tokens=False, return_tensors="pt",).to(model.device) with torch.inference_mode(): output = model.generate( **inputs, max_new_tokens=128, do_sample=False, use_cache=True, ) # Decode only the newly generated tokens.prompt_length = inputs["input_ids"].shape[1]new_tokens = output[0][prompt_length:]response = tokenizer.decode(new_tokens, skip_special_tokens=True) print("Model response:")print(response) # Optional: parse JSON if the model returned valid JSON.try: result = json.loads(response) print("Bounding box:", result.get("bbox")) print("Click point:", result.get("point"))except json.JSONDecodeError: print("The response was not valid JSON; inspect the raw output.")
Compatibility note: Unsloth's vision model loading and processor
APIs can vary by model and package version. Test this snippet against
the exact exported checkpoint and installed Unsloth version. If the
model requires a model-specific processor API, use the loading/inference
pattern from the original training notebook.
Kaggle: upload your own image and test
This optional example is intended for a Kaggle notebook after the model
and tokenizer have been loaded. It uses a file-upload widget; widget
availability can depend on the notebook environment.
from io import BytesIOfrom PIL import Imagefrom IPython.display import displayimport ipywidgets as widgets uploader = widgets.FileUpload(accept="image/*", multiple=False)display(uploader)
After uploading an image, run:
# Supports common ipywidgets 7 and 8 return formats.if not uploader.value: raise ValueError("Upload an image first.") if isinstance(uploader.value, dict): uploaded_file = next(iter(uploader.value.values()))else: uploaded_file = uploader.value[0] image_bytes = uploaded_file["content"]image = Image.open(BytesIO(image_bytes)).convert("RGB")display(image) instruction = input("Enter a UI instruction: ")
Then run the inference code in the previous section using this image
and instruction.
Visualize the predicted box
After the inference code has produced response, you can draw the
predicted bounding box and point:
import jsonimport refrom PIL import ImageDrawimport matplotlib.pyplot as plt match = re.search(r"\{[\s\S]*\}", response)if not match: raise ValueError("No JSON object found in the model response.") result = json.loads(match.group())bbox = result.get("bbox")point = result.get("point") if not (isinstance(bbox, list) and len(bbox) == 4): raise ValueError("Expected bbox: [x1, y1, x2, y2].") preview = image.copy()draw = ImageDraw.Draw(preview)draw.rectangle(bbox, outline="red", width=4) if isinstance(point, list) and len(point) == 2: x, y = point r = 6 draw.ellipse((x-r, y-r, x+r, y+r), fill="lime", outline="black", width=2) plt.figure(figsize=(14, 8))plt.imshow(preview)plt.axis("off")plt.show()
The visualization assumes the predicted coordinates are in the original
screenshot's pixel coordinate system. Verify the training labels and
preprocessing pipeline before relying on that assumption.
Publishing to Hugging Face
Use a Hugging Face access token with permission to create or write to
your model repository. Do not hard-code a token into a notebook, script,
or public repository.
Recommended: publish a merged model
In the training notebook, after training and before the runtime ends:
from huggingface_hub import loginimport os # Recommended: set HF_TOKEN as a Kaggle Secret or environment variable.# Never paste a real token into a public notebook.login(token=os.environ["HF_TOKEN"]) REPO_ID = "Navanjana/AgentEye-Grounder-0.8B-v1-GGUF" model.push_to_hub_merged( REPO_ID, tokenizer, token=os.environ["HF_TOKEN"], # Use the save/quantization options supported by your installed Unsloth version.)
Check Unsloth's current export documentation for the correct merged
precision and supported options for this particular vision model. A
merged 16-bit export is a practical choice for a broadly compatible
initial release, but it can require substantial RAM, disk space, and
upload time.
Alternative: publish a GGUF export
GGUF support and quantization options depend on the model architecture
and Unsloth version. Use this only after verifying that the trained
vision-language model's architecture and multimodal components are
supported by the selected GGUF export path.
Example pattern from Unsloth (verify current compatibility first):
model.push_to_hub_gguf( "Navanjana/AgentEye-Grounder-0.8B-v1-GGUF", tokenizer, quantization_method="q4_k_m", token=os.environ["HF_TOKEN"],)
Do not assume a GGUF export preserves all vision capabilities unless you
have tested image input end to end.
Limitations and responsible use
- Experimental release: this is the first iteration, not a
validated production-grade UI automation model.
- No published benchmark yet: no measured grounding accuracy, IoU,
click-point accuracy, latency, or cross-application evaluation was
supplied.
- Coordinate errors are possible: always validate the predicted
box and point before taking an action.
- UI changes affect results: themes, scaling, resolution,
localization, dialogs, and application updates may affect
predictions.
- Instruction ambiguity matters: instructions should identify a
specific visible target.
- JSON is not guaranteed: the model may occasionally return
malformed JSON or coordinates outside the image bounds.
- Safety boundary: this model predicts a location; it should not
be given unchecked authority to perform sensitive actions such as
payments, account changes, deleting files, or submitting forms.
- Privacy: screenshots may contain personal or confidential
information. Handle them according to the user's privacy
requirements and avoid uploading them to third-party services
without permission.
For an executor, validate that 0 <= x1 < x2 <= image_width and
0 <= y1 < y2 <= image_height, that the click point is in bounds, and
that the target is consistent with the instruction before interacting
with the operating system.
Roadmap
Planned direction for future versions:
- Train longer and tune the data pipeline based on evaluation results.
- Build a held-out benchmark for bounding-box IoU, point accuracy,
JSON validity, and inference latency.
- Test across multiple applications, display resolutions, scaling
factors, and UI themes.
- Improve robustness to ambiguous instructions, small icons, and
visually similar controls.
- Compare base, adapter, and merged checkpoints consistently.
- Integrate predictions into Agent's EYE / Desk Agent behind a
verification layer.
- Publish reproducible training configuration and evaluation results
with later releases.
These are future goals, not capabilities claimed for this v1 checkpoint.
Citation
If you use this model, cite the model repository once it is published
and cite the original dataset:
@misc{agenteye_grounder_v1, title = {AgentEye Grounder 0.8B v1}, author = {Navanjana}, year = {2026}, howpublished = {Hugging Face model repository}, note = {Experimental vision-language model for desktop UI grounding}}
Dataset:
Salesforce/grounding_dataset
Acknowledgements
AgentEye Grounder · Version 1
Small visual grounding models for more capable desktop agents.