jiangxuehaokeai

slide-grounding-mai-ui-2b-rl

Why This Model

The goal was to train a small 2B slide-grounding model that can compete with larger existing grounding models on a slide-specific benchmark.

Full test1 results below use 130,166 annotated slide items. Scores are assigned by Qwen3-VL on a 0..4 scale, where 0 means invalid output and 4 means precise manual-quality localization.

Table with columns: Model / setting, Params class, Prompt, Best bbox postprocessing, Avg score, Valid rate
Model / setting	Params class	Prompt	Best bbox postprocessing	Avg score	Valid rate
This model	2B	no thinking	none	3.7250	99.9962%
This model	2B	no thinking	loose	3.7291	99.9962%
`ui-venus-30b-fp8`	30B	direct	loose	3.722	99.62%
`ui-venus-8b-fp8`	8B	direct	loose	3.717	99.71%
`mai-ui-8b-fp8`	8B	thinking	loose	3.669	98.86%
`mai-ui-2b-fp8-no-think`	2B	no thinking	loose	3.185	88.01%
`mai-ui-8b-fp8-no-think`	8B	no thinking	loose	2.997	80.81%
`mai-ui-2b-fp8`	2B	thinking	loose	1.932	53.47%
`ui-venus-2b-fp8`	2B	direct	none	1.074	29.54%

The exported 2B model is slightly above the strongest evaluated off-the-shelf grounding baseline under the same Qwen3-VL evaluation protocol.

Evaluation Protocol

Evaluation uses the held-out test1 split of the prepared Zenodo10K-derived slide dataset.

Table with columns: Metric, Value
Metric	Value
Test slides attempted	9,569
Successfully annotated test slides	9,422
Test annotation failure rate	1.54%
Evaluated annotation items	130,166
Final checkpoint valid-box rate	99.9962%
Final checkpoint average score, raw bbox	3.7250 / 4
Final checkpoint average score, loose bbox postprocessing	3.7291 / 4

Scoring rubric:

Table with columns: Score, Meaning
Score	Meaning
0	Invalid output or invalid box format.
1	Wrong or mostly unusable localization.
2	Partially related but not a reliable target box.
3	Correct target with approximate boundaries.
4	Correct target with precise manual-quality boundaries.

This is a model-as-judge research metric, not a final human benchmark.

Bbox postprocessing:

Table with columns: Strategy, Meaning
Strategy	Meaning
`none`	Use the raw bounding box predicted by the grounding model.
`loose`	Recenter the box on the predicted point and expand each side to the larger original half-width/half-height, so the final box is looser but still centered.
`tight`	Recenter the box on the predicted point and make width and height both use the shorter side, producing a tighter square-like box.
`mean`	Recenter the box on the predicted point and make width and height both use the average of the original half-width and half-height.

Experiment Roadmap

The full training plan was deliberately incremental. Each row kept the best setting from previous rows unless the row explicitly tested an alternative.

Table with columns: Order, Stage, What changed, Best full-test score
Order	Stage	What changed	Best full-test score
1	SFT	train1, top-1 pseudo labels, score=4 only, no reasoning	3.6784
2	SFT	train1, top-3 x 4-postprocessing ensemble, score=4 only, no reasoning	3.7022
3	SFT	train1, ensemble, keep pseudo labels with scores 1-4, no reasoning	3.7050
4	SFT	train1, ensemble, scores 1-4, add MAI-8B reasoning text

Reward definitions:

Table with columns: Reward, Definition
Reward	Definition
Qwen score	Qwen evaluation score divided by `4`; invalid model outputs receive `0`.
Pseudo-label mean IoU	Mean IoU between the generated box and all score-4 boxes from the teacher pseudo-label pool.
Center consistency	`max(0, 1 - distance(point, box_center) / half_box_diagonal)` using the generated box itself.

Performance Figures

Best checkpoint from each experiment:

Best run scores

Checkpoint score curves:

Checkpoint score curves

Per-item-type comparison between the retained SFT baseline and the exported RL checkpoint:

Per type scores

The figure-generation script and the source CSV files are included in this model repository under scripts/ and figures/.

Data Preparation

The data comes from Zenodo10K slide decks. The preparation pipeline was:

Filter Zenodo10K decks and render slide screenshots.
Ask Qwen3-VL to annotate visible, semantically meaningful slide items.
Keep only legal annotation item types: chart, chart_component, image, image_component, reference_link, table, table_cell, table_column, table_row, and text_line.
Run multiple grounding models for every annotation item.
Evaluate each grounded candidate with Qwen3-VL.
Select pseudo labels for SFT from the best teacher candidates.

Annotation scale:

Table with columns: Split group, Attempted slides, Successful slides, Failed slides, Failure rate, Items
Split group	Attempted slides	Successful slides	Failed slides	Failure rate	Items
`test1`	9,569	9,422	147	1.54%	130,166
`train1..train9`	86,516	85,253	1,263	1.46%	1,162,921

Teacher Pseudo Labels

The SFT pseudo-label source was a top-3 x four-strategy ensemble.

Teacher models:

ui-venus-30b-fp8;
ui-venus-8b-fp8;
mai-ui-8b-fp8.

For every teacher output, we evaluated four bbox postprocessing strategies:

raw none;
loose;
tight;
mean.

For each item, the selected pseudo label was the candidate with the highest Qwen evaluation score.

Table with columns: Selector, Avg score, Score-4 rate, Score <= 2 count
Selector	Avg score	Score-4 rate	Score <= 2 count
Best single teacher: `ui-venus-30b-fp8 + loose`	3.722	76.53%	3,284
Top-3 teachers + `loose` only	3.915	92.54%	891
Top-3 teachers + `none/loose`	3.960	96.69%	582

The same selector was run on train1..train9 and reached about 3.982 average score with about 98.49% score-4 rate.

SFT Results

SFT used full-parameter HuggingFace/DDP training. LoRA was not used.

Main findings:

Ensemble pseudo labels were better than direct top-1 pseudo labels.
Keeping pseudo labels with scores 1..4 was better than keeping only score 4.
Adding MAI-8B reasoning text did not improve the 2B student.
Scaling from train1 to train1..train9 improved the best SFT score.

Best SFT checkpoints:

Table with columns: Run, Best checkpoint, Avg score, Valid rate
Run	Best checkpoint	Avg score	Valid rate
SFT train1, top-1 labels, score=4, no reasoning	`080p`	3.6784	99.45%
SFT train1, top-3 x 4 ensemble, score=4, no reasoning	`060p`	3.7022	99.95%
SFT train1, top-3 x 4 ensemble, score=1-4, no reasoning	`100p`	3.7050	99.95%

Measured formal SFT cost through the retained train1-9 baseline was about 372.17 productive GPU-hours on 8 x H100 GPUs.

RL Results

RL used GRPO from the retained SFT baseline.

Full-scale RL setup:

Table with columns: Field, Value
Field	Value
RL examples	1,145,400
Batch / grad accumulation / generations	2 / 2 / 4
Effective prompts per optimizer step	32
Learning rate	`1e-6`
KL beta	`0`
Max steps	35,794
Selected checkpoint	21,474 steps (`060p`)

Final train1-9 RL checkpoint curve:

Table with columns: Checkpoint, Avg score, Valid rate
Checkpoint	Avg score	Valid rate
`020p`	3.7232	99.9915%
`040p`	3.7217	99.9885%
`060p`	3.7250	99.9962%
`080p`	3.7238	99.9915%

The best checkpoint was not the final checkpoint, so intermediate checkpoint evaluation mattered.

Per-type score for the exported 060p checkpoint:

Table with columns: Item type, Count, Avg score, Delta vs SFT baseline
Item type	Count	Avg score	Delta vs SFT baseline
`chart`	1,612	3.8883	-0.0025
`chart_component`	4,359	3.6082	+0.0218
`image`	11,305	3.9010	-0.0047

Usage

Example vLLM serve command:

bash
vllm serve . \
  --served-model-name slide-grounding-mai-ui-2b-rl \
  --trust-remote-code \
  --dtype auto \
  --max-model-len 6144 \
  --limit-mm-per-prompt '{"image":1,"video":0}'

Recommended prompt:

text
You are a GUI grounding agent.

Given a slide screenshot and a text instruction or narration segment, locate
the visual region in the slide that the text is referring to.

Use normalized screenshot coordinates from 0 to 1000.
The origin (0,0) is the top-left corner of the image.
x increases to the right, y increases downward.

Return only valid JSON:
{"coordinate": [cx, cy], "bbox_2d": [x1, y1, x2, y2]}

Input instruction:
Find the item in the slide based on the description: "<description>".

The expected output is:

json
{"coordinate": [512, 208], "bbox_2d": [300, 160, 724, 256]}

Limitations

The reported metric is Qwen3-VL model-as-judge evaluation, not a final human benchmark.
The model is specialized for rendered slide screenshots and item descriptions similar to the annotation schema above.
The model predicts one box per instruction; it does not independently parse a full slide into all items.
The training labels are pseudo labels selected from stronger teacher models, not fully human-labeled boxes.
Check upstream Zenodo10K, MAI-UI, UI-Venus, and Qwen licenses before redistributing derived datasets or annotations.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

jiangxuehaokeai

Model Tree

Base

Tongyi-MAI/MAI-UI-2B

Fine-tuned

this model

Input Modalities

Text

Image

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Container

Explore FriendliAI today

Get started Talk to an engineer

Why This Model

The goal was to train a small 2B slide-grounding model that can compete with larger existing grounding models on a slide-specific benchmark.

Full test1 results below use 130,166 annotated slide items. Scores are assigned by Qwen3-VL on a 0..4 scale, where 0 means invalid output and 4 means precise manual-quality localization.

Table with columns: Model / setting, Params class, Prompt, Best bbox postprocessing, Avg score, Valid rate
Model / setting	Params class	Prompt	Best bbox postprocessing	Avg score	Valid rate
This model	2B	no thinking	none	3.7250	99.9962%
This model	2B	no thinking	loose	3.7291	99.9962%
`ui-venus-30b-fp8`	30B	direct	loose	3.722	99.62%
`ui-venus-8b-fp8`	8B	direct	loose	3.717	99.71%
`mai-ui-8b-fp8`	8B	thinking	loose	3.669	98.86%
`mai-ui-2b-fp8-no-think`	2B	no thinking	loose	3.185	88.01%
`mai-ui-8b-fp8-no-think`	8B	no thinking	loose	2.997	80.81%
`mai-ui-2b-fp8`	2B	thinking	loose	1.932	53.47%
`ui-venus-2b-fp8`	2B	direct	none	1.074	29.54%

The exported 2B model is slightly above the strongest evaluated off-the-shelf grounding baseline under the same Qwen3-VL evaluation protocol.

Evaluation Protocol

Evaluation uses the held-out test1 split of the prepared Zenodo10K-derived slide dataset.

Table with columns: Metric, Value
Metric	Value
Test slides attempted	9,569
Successfully annotated test slides	9,422
Test annotation failure rate	1.54%
Evaluated annotation items	130,166
Final checkpoint valid-box rate	99.9962%
Final checkpoint average score, raw bbox	3.7250 / 4
Final checkpoint average score, loose bbox postprocessing	3.7291 / 4

Scoring rubric:

Table with columns: Score, Meaning
Score	Meaning
0	Invalid output or invalid box format.
1	Wrong or mostly unusable localization.
2	Partially related but not a reliable target box.
3	Correct target with approximate boundaries.
4	Correct target with precise manual-quality boundaries.

This is a model-as-judge research metric, not a final human benchmark.

Bbox postprocessing:

Table with columns: Strategy, Meaning
Strategy	Meaning
`none`	Use the raw bounding box predicted by the grounding model.
`loose`	Recenter the box on the predicted point and expand each side to the larger original half-width/half-height, so the final box is looser but still centered.
`tight`	Recenter the box on the predicted point and make width and height both use the shorter side, producing a tighter square-like box.
`mean`	Recenter the box on the predicted point and make width and height both use the average of the original half-width and half-height.

Experiment Roadmap

The full training plan was deliberately incremental. Each row kept the best setting from previous rows unless the row explicitly tested an alternative.

Table with columns: Order, Stage, What changed, Best full-test score
Order	Stage	What changed	Best full-test score
1	SFT	train1, top-1 pseudo labels, score=4 only, no reasoning	3.6784
2	SFT	train1, top-3 x 4-postprocessing ensemble, score=4 only, no reasoning	3.7022
3	SFT	train1, ensemble, keep pseudo labels with scores 1-4, no reasoning	3.7050
4	SFT	train1, ensemble, scores 1-4, add MAI-8B reasoning text

Reward definitions:

Table with columns: Reward, Definition
Reward	Definition
Qwen score	Qwen evaluation score divided by `4`; invalid model outputs receive `0`.
Pseudo-label mean IoU	Mean IoU between the generated box and all score-4 boxes from the teacher pseudo-label pool.
Center consistency	`max(0, 1 - distance(point, box_center) / half_box_diagonal)` using the generated box itself.

Performance Figures

Best checkpoint from each experiment:

Best run scores

Checkpoint score curves:

Checkpoint score curves

Per-item-type comparison between the retained SFT baseline and the exported RL checkpoint:

Per type scores

The figure-generation script and the source CSV files are included in this model repository under scripts/ and figures/.

Data Preparation

The data comes from Zenodo10K slide decks. The preparation pipeline was:

Filter Zenodo10K decks and render slide screenshots.
Ask Qwen3-VL to annotate visible, semantically meaningful slide items.
Keep only legal annotation item types: chart, chart_component, image, image_component, reference_link, table, table_cell, table_column, table_row, and text_line.
Run multiple grounding models for every annotation item.
Evaluate each grounded candidate with Qwen3-VL.
Select pseudo labels for SFT from the best teacher candidates.

Annotation scale:

Table with columns: Split group, Attempted slides, Successful slides, Failed slides, Failure rate, Items
Split group	Attempted slides	Successful slides	Failed slides	Failure rate	Items
`test1`	9,569	9,422	147	1.54%	130,166
`train1..train9`	86,516	85,253	1,263	1.46%	1,162,921

Teacher Pseudo Labels

The SFT pseudo-label source was a top-3 x four-strategy ensemble.

Teacher models:

ui-venus-30b-fp8;
ui-venus-8b-fp8;
mai-ui-8b-fp8.

For every teacher output, we evaluated four bbox postprocessing strategies:

raw none;
loose;
tight;
mean.

For each item, the selected pseudo label was the candidate with the highest Qwen evaluation score.

Table with columns: Selector, Avg score, Score-4 rate, Score <= 2 count
Selector	Avg score	Score-4 rate	Score <= 2 count
Best single teacher: `ui-venus-30b-fp8 + loose`	3.722	76.53%	3,284
Top-3 teachers + `loose` only	3.915	92.54%	891
Top-3 teachers + `none/loose`	3.960	96.69%	582

The same selector was run on train1..train9 and reached about 3.982 average score with about 98.49% score-4 rate.

SFT Results

SFT used full-parameter HuggingFace/DDP training. LoRA was not used.

Main findings:

Ensemble pseudo labels were better than direct top-1 pseudo labels.
Keeping pseudo labels with scores 1..4 was better than keeping only score 4.
Adding MAI-8B reasoning text did not improve the 2B student.
Scaling from train1 to train1..train9 improved the best SFT score.

Best SFT checkpoints:

Table with columns: Run, Best checkpoint, Avg score, Valid rate
Run	Best checkpoint	Avg score	Valid rate
SFT train1, top-1 labels, score=4, no reasoning	`080p`	3.6784	99.45%
SFT train1, top-3 x 4 ensemble, score=4, no reasoning	`060p`	3.7022	99.95%
SFT train1, top-3 x 4 ensemble, score=1-4, no reasoning	`100p`	3.7050	99.95%

Measured formal SFT cost through the retained train1-9 baseline was about 372.17 productive GPU-hours on 8 x H100 GPUs.

RL Results

RL used GRPO from the retained SFT baseline.

Full-scale RL setup:

Table with columns: Field, Value
Field	Value
RL examples	1,145,400
Batch / grad accumulation / generations	2 / 2 / 4
Effective prompts per optimizer step	32
Learning rate	`1e-6`
KL beta	`0`
Max steps	35,794
Selected checkpoint	21,474 steps (`060p`)

Final train1-9 RL checkpoint curve:

Table with columns: Checkpoint, Avg score, Valid rate
Checkpoint	Avg score	Valid rate
`020p`	3.7232	99.9915%
`040p`	3.7217	99.9885%
`060p`	3.7250	99.9962%
`080p`	3.7238	99.9915%

The best checkpoint was not the final checkpoint, so intermediate checkpoint evaluation mattered.

Per-type score for the exported 060p checkpoint:

Table with columns: Item type, Count, Avg score, Delta vs SFT baseline
Item type	Count	Avg score	Delta vs SFT baseline
`chart`	1,612	3.8883	-0.0025
`chart_component`	4,359	3.6082	+0.0218
`image`	11,305	3.9010	-0.0047

Usage

Example vLLM serve command:

bash
vllm serve . \
  --served-model-name slide-grounding-mai-ui-2b-rl \
  --trust-remote-code \
  --dtype auto \
  --max-model-len 6144 \
  --limit-mm-per-prompt '{"image":1,"video":0}'

Recommended prompt:

text
You are a GUI grounding agent.

Given a slide screenshot and a text instruction or narration segment, locate
the visual region in the slide that the text is referring to.

Use normalized screenshot coordinates from 0 to 1000.
The origin (0,0) is the top-left corner of the image.
x increases to the right, y increases downward.

Return only valid JSON:
{"coordinate": [cx, cy], "bbox_2d": [x1, y1, x2, y2]}

Input instruction:
Find the item in the slide based on the description: "<description>".

The expected output is:

json
{"coordinate": [512, 208], "bbox_2d": [300, 160, 724, 256]}

Limitations

The reported metric is Qwen3-VL model-as-judge evaluation, not a final human benchmark.
The model is specialized for rendered slide screenshots and item descriptions similar to the annotation schema above.
The model predicts one box per instruction; it does not independently parse a full slide into all items.
The training labels are pseudo labels selected from stronger teacher models, not fully human-labeled boxes.
Check upstream Zenodo10K, MAI-UI, UI-Venus, and Qwen licenses before redistributing derived datasets or annotations.

slide-grounding-mai-ui-2b-rl

README

Why This Model

Evaluation Protocol

Experiment Roadmap

Performance Figures

Data Preparation

Teacher Pseudo Labels

SFT Results

RL Results

Usage

Limitations

Explore FriendliAI today

README

Why This Model

Evaluation Protocol

Experiment Roadmap

Performance Figures

Data Preparation

Teacher Pseudo Labels

SFT Results

RL Results

Usage

Limitations