GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
docling-project
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
docling-project
Model Tree
Input Modalities
Output Modalities
Supported Functionality
May 2026: Updated main with the ScreenVLM checkpoint trained on ScreenParse v2. This release uses the v2 training data with more robust quality filtering, 1,447,100 high-quality screenshots, and varied viewport resolutions. The original ScreenVLM checkpoint trained on ScreenParse v1 is retained on the v1 branch.
Developed by: IBM Research Zurich - ETH Zurich
Model type: Multi-modal model (image+text-to-text)
Language(s): English
License: Apache 2.0
Paper: ScreenParse: Moving Beyond Sparse Grounding with Complete Screen Parsing
Dataset: docling-project/screenparse
ScreenVLM builds upon the Idefics3 architecture with siglip2-base-patch16-512 as the vision encoder and a Granite 165M LLM as the language backbone. The current main checkpoint was trained on ScreenParse v2 full-element screen parsing supervision across 55 semantic UI classes.
ScreenVLM generates output in ScreenTag format, where each UI element is wrapped in semantic tags with location tokens:
html
<screentag><button><loc_10><loc_20><loc_50><loc_35>Submit</button><link><loc_100><loc_200><loc_180><loc_210>Learn more</link><navigation_bar><loc_0><loc_0><loc_500><loc_30> <link><loc_10><loc_5><loc_60><loc_25>Home</link> <link><loc_70><loc_5><loc_120><loc_25>About</link></navigation_bar></screentag>Each <loc_X> token represents a coordinate in the normalized [0, 500] space. Four consecutive location tokens define <left><top><right><bottom> of the bounding box.
python
import reimport torchfrom transformers import AutoProcessor, AutoModelForVision2Seqfrom transformers.image_utils import load_image DEVICE = "cuda" if torch.cuda.is_available() else "cpu"MODEL_PATH = "docling-project/ScreenVLM"NORM_SIZE = 500 image = load_image("https://example.com/screenshot.png") processor = AutoProcessor.from_pretrained(MODEL_PATH)model = AutoModelForVision2Seq.from_pretrained( MODEL_PATH, torch_dtype=torch.bfloat16, _attn_implementation="flash_attention_2" if DEVICE == "cuda" else "sdpa",).to(DEVICE) messages = [ { "role": "user", "content": [ {"type": "image"}, {"type": "text", "text": "Generate the screen representation for this UI:"}, ], },] prompt = processor.apply_chat_template(messages, add_generation_prompt=True)inputs = processor(text=prompt, images=[image], return_tensors="pt").to(DEVICE) generated_ids = model.generate(**inputs, max_new_tokens=6192)prompt_length = inputs.input_ids.shape[1]output = processor.batch_decode( generated_ids[:, prompt_length:], skip_special_tokens=False,)[0].lstrip() def parse_screentag(text, width, height): pattern = re.compile( r"<(?P<tag>[a-zA-Z][a-zA-Z0-9_]*)>" r"\s*<loc_(?P<l>\d+)><loc_(?P<t>\d+)><loc_(?P<r>\d+)><loc_(?P<b>\d+)>" r"(?P<text>[^<]*)" ) elements = [] for m in pattern.finditer(text): l, t, r, b = [max(0, min(int(m.group(k)), NORM_SIZE)) for k in ("l", "t", "r", "b")] if r < l: l, r = r, l if b < t: t, b = b, t x = l / NORM_SIZE * width y = t / NORM_SIZE * height w = (r - l) / NORM_SIZE * width h = (b - t) / NORM_SIZE * height elements.append({ "label": m.group("tag"), "bbox": (x, y, w, h), "text": m.group("text").strip() or None, }) return elements elements = parse_screentag(output, *image.size)for el in elements: print(f"{el['label']:20s} bbox=({int(el['bbox'][0]):4d},{int(el['bbox'][1]):4d},{int(el['bbox'][2]):4d},{int(el['bbox'][3]):4d}) text={el['text']!r}")python
import osimport reimport timefrom vllm import LLM, SamplingParamsfrom transformers import AutoProcessorfrom PIL import Image MODEL_PATH = "docling-project/ScreenVLM"IMAGE_DIR = "screenshots/"PROMPT_TEXT = "Generate the screen representation for this UI:"NORM_SIZE = 500 messages = [ { "role": "user", "content": [ {"type": "image"}, {"type": "text", "text": PROMPT_TEXT}, ], },] llm = LLM(model=MODEL_PATH, limit_mm_per_prompt={"image": 1})processor = AutoProcessor.from_pretrained(MODEL_PATH) sampling_params = SamplingParams( temperature=0.0, max_tokens=6192, skip_special_tokens=False,) batched_inputs = []image_sizes = []for img_file in sorted(os.listdir(IMAGE_DIR)): if img_file.lower().endswith((".png", ".jpg", ".jpeg")): img_path = os.path.join(IMAGE_DIR, img_file) image = Image.open(img_path).convert("RGB") prompt = processor.apply_chat_template(messages, add_generation_prompt=True) batched_inputs.append({"prompt": prompt, "multi_modal_data": {"image": image}}) image_sizes.append((img_file, image.size)) start = time.time()outputs = llm.generate(batched_inputs, sampling_params=sampling_params)print(f"Total: {time.time() - start:.1f}s for {len(batched_inputs)} images")ScreenVLM was trained using the nanoVLM framework with 16 NVIDIA H100 GPUs.
Training data: ScreenParse v2 full-element annotations for complete screen parsing. The v2 training data contains 1,447,100 high-quality web page screenshots across varied viewport resolutions with dense UI element supervision, including bounding boxes, semantic labels, text content, interactability flags, and reading order.
The original ScreenVLM checkpoint trained on ScreenParse v1 remains available with revision="v1".
bibtex
@misc{gurbuz2026movingsparsegroundingcomplete, title={ScreenParse: Moving Beyond Sparse Grounding with Complete Screen Parsing Supervision}, author={A. Said Gurbuz and Sunghwan Hong and Ahmed Nassar and Marc Pollefeys and Peter Staar}, year={2026}, eprint={2602.14276}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2602.14276},}import reimport torchfrom transformers import AutoProcessor, AutoModelForVision2Seqfrom transformers.image_utils import load_image DEVICE = "cuda" if torch.cuda.is_available() else "cpu"MODEL_PATH = "docling-project/ScreenVLM"NORM_SIZE = 500 image = load_image("https://example.com/screenshot.png") processor = AutoProcessor.from_pretrained(MODEL_PATH)model = AutoModelForVision2Seq.from_pretrained( MODEL_PATH, torch_dtype=torch.bfloat16, _attn_implementation="flash_attention_2" if DEVICE == "cuda" else "sdpa",).to(DEVICE) messages = [ { "role": "user", "content": [ {"type": "image"}, {"type": "text", "text": "Generate the screen representation for this UI:"}, ], },] prompt = processor.apply_chat_template(messages, add_generation_prompt=True)inputs = processor(text=prompt, images=[image], return_tensors="pt").to(DEVICE) generated_ids = model.generate(**inputs, max_new_tokens=6192)prompt_length = inputs.input_ids.shape[1]output = processor.batch_decode( generated_ids[:, prompt_length:], skip_special_tokens=False,)[0].lstrip() def parse_screentag(text, width, height): pattern = re.compile( r"<(?P<tag>[a-zA-Z][a-zA-Z0-9_]*)>" r"\s*<loc_(?P<l>\d+)><loc_(?P<t>\d+)><loc_(?P<r>\d+)><loc_(?P<b>\d+)>" r"(?P<text>[^<]*)" ) elements = [] for m in pattern.finditer(text): l, t, r, b = [max(0, min(int(m.group(k)), NORM_SIZE)) for k in ("l", "t", "r", "b")] if r < l: l, r = r, l if b < t: t, b = b, t x = l / NORM_SIZE * width y = t / NORM_SIZE * height w = (r - l) / NORM_SIZE * width h = (b - t) / NORM_SIZE * height elements.append({ "label": m.group("tag"), "bbox": (x, y, w, h), "text": m.group("text").strip() or None, }) return elements elements = parse_screentag(output, *image.size)for el in elements: print(f"{el['label']:20s} bbox=({int(el['bbox'][0]):4d},{int(el['bbox'][1]):4d},{int(el['bbox'][2]):4d},{int(el['bbox'][3]):4d}) text={el['text']!r}")import osimport reimport timefrom vllm import LLM, SamplingParamsfrom transformers import AutoProcessorfrom PIL import Image MODEL_PATH = "docling-project/ScreenVLM"IMAGE_DIR = "screenshots/"PROMPT_TEXT = "Generate the screen representation for this UI:"NORM_SIZE = 500 messages = [ { "role": "user", "content": [ {"type": "image"}, {"type": "text", "text": PROMPT_TEXT}, ], },] llm = LLM(model=MODEL_PATH, limit_mm_per_prompt={"image": 1})processor = AutoProcessor.from_pretrained(MODEL_PATH) sampling_params = SamplingParams( temperature=0.0, max_tokens=6192, skip_special_tokens=False,) batched_inputs = []image_sizes = []for img_file in sorted(os.listdir(IMAGE_DIR)): if img_file.lower().endswith((".png", ".jpg", ".jpeg")): img_path = os.path.join(IMAGE_DIR, img_file) image = Image.open(img_path).convert("RGB") prompt = processor.apply_chat_template(messages, add_generation_prompt=True) batched_inputs.append({"prompt": prompt, "multi_modal_data": {"image": image}}) image_sizes.append((img_file, image.size)) start = time.time()outputs = llm.generate(batched_inputs, sampling_params=sampling_params)print(f"Total: {time.time() - start:.1f}s for {len(batched_inputs)} images")@misc{gurbuz2026movingsparsegroundingcomplete, title={ScreenParse: Moving Beyond Sparse Grounding with Complete Screen Parsing Supervision}, author={A. Said Gurbuz and Sunghwan Hong and Ahmed Nassar and Marc Pollefeys and Peter Staar}, year={2026}, eprint={2602.14276}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2602.14276},}<screentag><button><loc_10><loc_20><loc_50><loc_35>Submit</button><link><loc_100><loc_200><loc_180><loc_210>Learn more</link><navigation_bar><loc_0><loc_0><loc_500><loc_30> <link><loc_10><loc_5><loc_60><loc_25>Home</link> <link><loc_70><loc_5><loc_120><loc_25>About</link></navigation_bar></screentag>