GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
ayushadarsh7
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
ayushadarsh7
Model Tree
Qwen/Qwen3-VL-4B-Instruct
Input Modalities
Output Modalities
Supported Functionality
A CUDA GPU is recommended. FP32 model weights alone occupy about 17.8 GB; allow additional memory for vision activations, KV cache, and generation. Actual inference memory requirements depend on the screenshot and generation length.
Download the helper and its dependencies:
bash
pip install 'huggingface_hub>=0.34,<1.0'hf download ayushadarsh7/kayak_v2 infer.py requirements.txt --local-dir kayak_v2_codepip install -r kayak_v2_code/requirements.txtInstall a PyTorch build suitable for your GPU/CUDA version if the default package does not match your machine. The tested environment uses Python 3.12, PyTorch 2.9.0+cu128, Transformers 4.57.1, and Accelerate 1.11.0.
Click example:
bash
python kayak_v2_code/infer.py \ --image /path/to/screenshot.png \ --instruction 'Click Search.'Fill example:
bash
python kayak_v2_code/infer.py \ --image /path/to/screenshot.png \ --instruction 'Enter New Delhi in the origin field.' \ --value 'New Delhi'To clear a field, pass --value '' with a clear-field instruction. Empty strings are intentional training targets.
The helper defaults to --model ayushadarsh7/kayak_v2 --device cuda:0 --dtype float32. It uses SDPA, greedy decoding, and up to 128 new tokens. It also applies the mathematically equivalent linear vision-patch projection used by the training project to avoid a slow Conv3d path on some Blackwell systems. It prints the prediction and does not click or type anything.
For lower memory use, --dtype bfloat16 is available, but rounding full weights can change small fine-tuning updates; it has not been separately benchmarked for this release. FP32 is the default for preserving the reconstructed weights. CPU execution is possible with --device cpu --dtype float32, but slower.
After downloading infer.py, from the kayak_v2_code directory:
python
from infer import Grounder grounder = Grounder('ayushadarsh7/kayak_v2', device='cuda:0', dtype='float32')prediction = grounder.predict( '/path/to/screenshot.png', 'Enter New Delhi in the origin field.', value='New Delhi',)print(prediction)Standard Transformers loading also works:
python
import torchfrom transformers import AutoProcessor, Qwen3VLForConditionalGeneration processor = AutoProcessor.from_pretrained( 'ayushadarsh7/kayak_v2', min_pixels=262144, max_pixels=524288,)model = Qwen3VLForConditionalGeneration.from_pretrained( 'ayushadarsh7/kayak_v2', dtype=torch.float32, attn_implementation='sdpa',).to('cuda:0').eval()Use the helper for the complete prompt/image preprocessing and generation procedure. The processor and chat template are shipped with the model.
The training prompt is:
text
Task: <instruction>Text to enter: "<value>"Respond with only JSON using action and the applicable bbox, value, or key fields. bbox is [x, y, width, height] on a 0-1000 scale relative to the image; x/y is the top-left corner.The Text to enter line is present only for fill actions; the value is JSON-escaped. The user message contains the screenshot followed by this text. Do not provide red-box overlays or after-action screenshots as grounding inputs.
Illustrative output (not a measured prediction):
json
{"action":"click","bbox":[100,200,150,50]}bbox is [x, y, width, height], normalized to 0–1000. Decimal coordinates are allowed. It is not [x1, y1, x2, y2] and is not in pixels. For an original image of width W and height H, the box center in pixels is:
python
x, y, w, h = prediction['bbox']click_x = (x + w / 2) * W / 1000click_y = (y + h / 2) * H / 1000Parse and validate generated JSON and image bounds before using it in an application. The inherited prompt mentions key, but this fine-tune's 728 eligible examples contain only click and fill actions; keyboard, scroll, drag, planner and checker behavior are not trained or evaluated by this release.
The source collection is kayak-v2-batch-2026-10-23: 40 recorded episodes and 973 actions. Its filtered clicker export contains 728 grounding examples from 39 usable episodes: 626 clicks and 102 fills, including 44 field-clearing fills.
| Product | Train episodes | Test episodes | Train examples | Test examples |
|---|---|---|---|---|
| Flights | 8 | 2 | 125 | 43 |
| Hotels / Stays | 8 | 2 | 155 | 38 |
| Car rental | 8 | 2 | 147 | 37 |
| Flight + Hotel packages | 7 | 2 | 154 | 29 |
| Total | 31 | 8 | 581 | 147 |
Whole episodes stay in one split. The allocation balances functional element categories and action types. All 20 recurring categories appear in both splits; hotel-name search, map-location search and result comparison occur in only one episode each and stay in training. All examples are used once, with no episode or byte-identical screenshot overlap.
There are nine flagged cross-split similar-image pairs, primarily recurring search-page layouts with different form values or menu states. All episodes came from one shared browser session. This is a within-session split, not evidence of independent-session or unseen-site generalization. Source records remain pending independent acceptance; collection review and structural validation are not human-approval guarantees. Raw data and screenshots are not distributed in this model repository.
| Setting | Value |
|---|---|
| Base | Qwen/Qwen3-VL-4B-Instruct |
| Pinned base revision | ebb281ec70b05090aa6165b016eac8ec08e71b17 |
| Method | Full-parameter SFT; assistant-answer-only loss |
| Trainable parameters | Approximately 4.438 billion |
| Epochs / optimizer steps | 4 / 148 |
| Seed | 1 |
| Learning rate | 5e-6 |
| Optimizer | AdamW, betas (0.9, 0.999), epsilon 1e-8 |
| Scheduler | 3% warmup, then cosine decay |
| Effective batch | 16 (micro-batch 4 × 4 accumulation passes) |
| Weight decay / gradient clipping | 0 / 1.0 |
| Precision | FP32 master weights, BF16 autocast compute |
| Attention / memory | SDPA; gradient checkpointing |
| Image pixels | 262,144 minimum; 524,288 maximum |
| Maximum training sequence | 2,048 tokens |
| GPU | NVIDIA RTX PRO 6000 Blackwell Server Edition |
| Training duration | 567.8 seconds |
| Peak allocated training GPU memory | 89.4 GB |
| Aggregate training loss | 0.535825 |
The learning rate and four-epoch schedule were inherited from a previous study run, not selected using this V2 test set. Settings and checkpoint provenance are also in training_metadata.json.
Evaluated the frozen 147-example / 8-episode test split once, using the published FP32 model, deterministic SDPA with TF32 enabled, greedy decoding, and 128 generated tokens. The token limit matches the published inference helper; older study evaluations used 64. Grounding success requires the correct action, valid box, predicted center inside the target, and exact fill text when applicable. Scenario status covers independent recorded screenshots, not live end-to-end workflow success.
| Metric | Held-out result |
|---|---|
| Grounding + action success | 137/147 (93.20%) |
| Box-center hit rate | 93.88% |
| Action accuracy | 99.32% |
| Parsed / strict JSON | 100.00% / 100.00% |
| Valid bounding boxes | 100.00% |
| Exact fill value | 25/25 (100.00%) |
| Mean / median IoU | 0.796108 / 0.893316 |
| IoU >= 0.50 / >= 0.75 | 88.44% / 74.15% |
| Assistant-token loss / perplexity | 0.502382 / 1.652653 |
| Episodes with every evaluated step passing | 4/8 |
| Generation limit reached | 0 |
| Press-key exact match | N/A: no press examples |
| Episode | Product | Passed / evaluated | Failed original steps |
|---|---|---|---|
| b08-packages-august | packages | 19/19 | None |
| b09-flights-airline | flights | 16/19 | 18, 19, 21 |
| b11-cars-search | cars | 13/13 | None |
| b12-packages-search | packages | 10/10 | None |
| b21-flights-multicity | flights | 20/24 | 13, 19, 21, 22 |
| b26-stays-named-gallery | stays | 12/14 | 2, 4 |
| b27-cars-van-passengers | cars | 24/24 | None |
| b34-stays-stay-total-budget | stays | 23/24 | 19 |
The separate scenario_results.md report lists every evaluated task with its original step number and PASS/FAIL status. For each failure it includes the expected JSON, actual model output and failure reason. See scenario_results.json for structured results and failed_steps.jsonl for failures only.
Full metrics: evaluation_results.md, summary.json, and metrics_by_group.json. Every prediction is saved in predictions.jsonl; protocol.json records settings and hashes; verification.json records independent rescoring and coverage checks.
All 147 expected IDs were evaluated exactly once. Original labels, tasks and step numbers were checked against the frozen source, every prediction was rescored, and scenario totals reconciled with overall metrics. The shared-session and near-image limitations still apply; these numbers are not an unseen-site or live-workflow benchmark.
Held-out evaluation is now complete. The 147-example test set was not used during training or checkpoint selection. The results below are for this model; the training loss above and previous Kayak-model results are separate.
All converted examples passed the actual training collator's token-prefix, answer-mask and sequence-length checks. The exported model and inference helper passed a generation smoke check on training examples; that checks loading and output wiring, not generalization. See inference_validation.json for the recorded result.
The model is specialized to the captured Kayak UI and English instructions. Layout, locale, content, scaling and website changes may affect performance. It may produce invalid JSON, incorrect boxes or incorrect text. It has not been evaluated for end-to-end browser tasks, other websites, or safety-critical actions.
Apache-2.0; see LICENSE. The base model is Qwen3-VL-4B-Instruct, whose upstream model card identifies the same license. This model is a community fine-tune and is not affiliated with or endorsed by Kayak or Qwen.
pip install 'huggingface_hub>=0.34,<1.0'hf download ayushadarsh7/kayak_v2 infer.py requirements.txt --local-dir kayak_v2_codepip install -r kayak_v2_code/requirements.txtpython kayak_v2_code/infer.py \ --image /path/to/screenshot.png \ --instruction 'Click Search.'python kayak_v2_code/infer.py \ --image /path/to/screenshot.png \ --instruction 'Enter New Delhi in the origin field.' \ --value 'New Delhi'from infer import Grounder grounder = Grounder('ayushadarsh7/kayak_v2', device='cuda:0', dtype='float32')prediction = grounder.predict( '/path/to/screenshot.png', 'Enter New Delhi in the origin field.', value='New Delhi',)print(prediction)import torchfrom transformers import AutoProcessor, Qwen3VLForConditionalGeneration processor = AutoProcessor.from_pretrained( 'ayushadarsh7/kayak_v2', min_pixels=262144, max_pixels=524288,)model = Qwen3VLForConditionalGeneration.from_pretrained( 'ayushadarsh7/kayak_v2', dtype=torch.float32, attn_implementation='sdpa',).to('cuda:0').eval()Task: <instruction>Text to enter: "<value>"Respond with only JSON using action and the applicable bbox, value, or key fields. bbox is [x, y, width, height] on a 0-1000 scale relative to the image; x/y is the top-left corner.{"action":"click","bbox":[100,200,150,50]}x, y, w, h = prediction['bbox']click_x = (x + w / 2) * W / 1000click_y = (y + h / 2) * H / 1000