Srinjoy99

tadpole-0.8b

README

License: apache-2.0

Example

A hand-drawn UML activity diagram from the CAS2UML test split (CC BY 4.0). The model never saw this image in training.

Tadpole's answer, exactly right:

text

diagram activity dir=TBnode n1 initialnode n2 action "Sign Contract"node n3 forknode n4 action "IT: Provision Laptop"node n5 action "HR: Setup Benefits"node n6 action "Legal: NDA Signature"node n7 joinnode n8 action "First Day Orientation"node n9 finaledge n1 n2edge n2 n3edge n3 n4edge n3 n5edge n3 n6edge n4 n7edge n5 n7edge n6 n7edge n7 n8edge n8 n9

Ask for boxes and every element also gets a position (0–1000 coordinates). Here they are drawn back onto a photo of a hand-drawn BPMN diagram (hdBPMN test split, CC BY 4.0):

What it reads

Flowcharts, UML activity, sequence, class and state diagrams, BPMN processes, ER diagrams, logic circuits (gates and wires), and generic node-link graphs such as mind maps and architecture diagrams.

The output names each element's kind, such as decision, fork, interface or nand, and its text. It also gives every arrow with its direction and label, and containers such as lanes, pools, packages and composite states. Boxes are optional.

Results

These are the test-set results on 580 images from 12 sources. "Never trained" marks sources kept out of training, so their style is completely new to the model.

Table with columns: Source, Style, Edge F1, Node F1
Source	Style	Edge F1	Node F1
FlowVQA	rendered flowcharts	0.988	0.998
CAS2UML	hand-drawn UML	0.958	0.992
Image-to-Mermaid v3	rendered, 6 diagram types	0.946	0.994
rakitha (never trained)	rendered flowcharts	0.921	0.955
DiDi	hand-drawn ink flowcharts	0.917	0.968
FlowGen	4 renderers, scanned copies

Edge F1 counts arrows with the right endpoints and direction.
Node F1 counts the elements.
Valid output (it parses without errors) is 99.3%. 36% of answers are exactly right, character for character.

Use it with transformers

python

import torchfrom PIL import Imagefrom transformers import AutoModelForImageTextToText, AutoProcessor repo = "Srinjoy99/tadpole-0.8b"processor = AutoProcessor.from_pretrained(repo)model = AutoModelForImageTextToText.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto") image = Image.open("diagram.png").convert("RGB")messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "Convert this diagram to DGM."}]}]prompt = processor.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False)inputs = processor(text=[prompt], images=[image], return_tensors="pt",                   images_kwargs={"size": {"shortest_edge": 65536, "longest_edge": 1254400}}).to(model.device) output = model.generate(**inputs, max_new_tokens=1536, do_sample=False)dgm = processor.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)print(dgm)

Required settings:
- enable_thinking=False: the model was trained with an empty think block.
- Greedy decoding.
- The image size limits shown: 64 to 1,225 image tokens, as in training.
Boxes: use the prompt "Convert this diagram to DGM with boxes.".
Transparent PNGs: flatten them onto white first, so black lines stay visible.

JSON, YAML or XML

The answer is DGM, the compact line format of the Diagram IR. The diagram_ir package in the GitHub repo converts it losslessly:

bash

python -m diagram_ir convert diagram.dgm --to json,yaml,xml -o exports/

Or, from Python:

python

from diagram_ir import dump, parse diagram, problems = parse(dgm)  # lenient: a cut-off answer still keeps every complete lineprint(dump(diagram, "json"))

The repo also has a one-line CLI (python -m deploy.predict diagram.png --to json) and a demo notebook. The format is defined in SPEC.md.

How it was trained

Tadpole is a full fine-tune of Qwen3.5-0.8B on one RTX 3090, in about 24 GPU-hours:

three supervised stages;
two rounds of reinforcement learning (DAPO-style GRPO) whose reward is graph F1;
weight merging.

The training data comes from 12 sources with about 137,000 image–target pairs, including 26,000 synthetic diagrams generated for the project. Every target comes from the source's own structured ground truth, never from OCR. The full story, including a metric bug the RL run exploited and how it was fixed, is in TRAINING.md. The datasets and cleaning are in DATASETS.md.

Training data:

Apache-2.0: FlowGen
MIT: Image-to-Mermaid v3, Flowchart-to-Mermaid, FlowVQA
CC BY 4.0: DiDi, hdBPMN, LogicBench-1K, CAS2UML
Permissively licensed files only: The Stack v2 PlantUML

Sets whose licenses do not allow training were used for testing only.

Limitations

Big circuits. Circuits with 11 or more gates are mostly wrong (edge F1 0.27). Small ones do much better (0.71 for up to 6 gates).
Hand-drawn BPMN. Arrows into collapsed pools are often missing.
Handwriting. On never-trained handwritten flowcharts, 59% of node texts are exact.
Very large diagrams. Answers are capped at 1,536 tokens.
Boxes. They are reliable for BPMN and circuits, the styles that had box annotations in training.
Out of scope. Charts, plots and tables.

License

Apache-2.0, like the base model. The CC BY 4.0 training sets are credited above.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Srinjoy99

Model Tree

Base

Qwen/Qwen3.5-0.8B

Fine-tuned

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Example

A hand-drawn UML activity diagram from the CAS2UML test split (CC BY 4.0). The model never saw this image in training.

Tadpole's answer, exactly right:

text

diagram activity dir=TBnode n1 initialnode n2 action "Sign Contract"node n3 forknode n4 action "IT: Provision Laptop"node n5 action "HR: Setup Benefits"node n6 action "Legal: NDA Signature"node n7 joinnode n8 action "First Day Orientation"node n9 finaledge n1 n2edge n2 n3edge n3 n4edge n3 n5edge n3 n6edge n4 n7edge n5 n7edge n6 n7edge n7 n8edge n8 n9

Ask for boxes and every element also gets a position (0–1000 coordinates). Here they are drawn back onto a photo of a hand-drawn BPMN diagram (hdBPMN test split, CC BY 4.0):

What it reads

Flowcharts, UML activity, sequence, class and state diagrams, BPMN processes, ER diagrams, logic circuits (gates and wires), and generic node-link graphs such as mind maps and architecture diagrams.

Results

These are the test-set results on 580 images from 12 sources. "Never trained" marks sources kept out of training, so their style is completely new to the model.

Table with columns: Source, Style, Edge F1, Node F1
Source	Style	Edge F1	Node F1
FlowVQA	rendered flowcharts	0.988	0.998
CAS2UML	hand-drawn UML	0.958	0.992
Image-to-Mermaid v3	rendered, 6 diagram types	0.946	0.994
rakitha (never trained)	rendered flowcharts	0.921	0.955
DiDi	hand-drawn ink flowcharts	0.917	0.968
FlowGen	4 renderers, scanned copies

Edge F1 counts arrows with the right endpoints and direction.
Node F1 counts the elements.
Valid output (it parses without errors) is 99.3%. 36% of answers are exactly right, character for character.

Use it with transformers

python

import torchfrom PIL import Imagefrom transformers import AutoModelForImageTextToText, AutoProcessor repo = "Srinjoy99/tadpole-0.8b"processor = AutoProcessor.from_pretrained(repo)model = AutoModelForImageTextToText.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto") image = Image.open("diagram.png").convert("RGB")messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "Convert this diagram to DGM."}]}]prompt = processor.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False)inputs = processor(text=[prompt], images=[image], return_tensors="pt",                   images_kwargs={"size": {"shortest_edge": 65536, "longest_edge": 1254400}}).to(model.device) output = model.generate(**inputs, max_new_tokens=1536, do_sample=False)dgm = processor.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)print(dgm)

Required settings:
- enable_thinking=False: the model was trained with an empty think block.
- Greedy decoding.
- The image size limits shown: 64 to 1,225 image tokens, as in training.
Boxes: use the prompt "Convert this diagram to DGM with boxes.".
Transparent PNGs: flatten them onto white first, so black lines stay visible.

JSON, YAML or XML

The answer is DGM, the compact line format of the Diagram IR. The diagram_ir package in the GitHub repo converts it losslessly:

bash

python -m diagram_ir convert diagram.dgm --to json,yaml,xml -o exports/

Or, from Python:

python

from diagram_ir import dump, parse diagram, problems = parse(dgm)  # lenient: a cut-off answer still keeps every complete lineprint(dump(diagram, "json"))

The repo also has a one-line CLI (python -m deploy.predict diagram.png --to json) and a demo notebook. The format is defined in SPEC.md.

How it was trained

Tadpole is a full fine-tune of Qwen3.5-0.8B on one RTX 3090, in about 24 GPU-hours:

three supervised stages;
two rounds of reinforcement learning (DAPO-style GRPO) whose reward is graph F1;
weight merging.

Training data:

Apache-2.0: FlowGen
MIT: Image-to-Mermaid v3, Flowchart-to-Mermaid, FlowVQA
CC BY 4.0: DiDi, hdBPMN, LogicBench-1K, CAS2UML
Permissively licensed files only: The Stack v2 PlantUML

Sets whose licenses do not allow training were used for testing only.

Limitations

Big circuits. Circuits with 11 or more gates are mostly wrong (edge F1 0.27). Small ones do much better (0.71 for up to 6 gates).
Hand-drawn BPMN. Arrows into collapsed pools are often missing.
Handwriting. On never-trained handwritten flowcharts, 59% of node texts are exact.
Very large diagrams. Answers are capped at 1,536 tokens.
Boxes. They are reliable for BPMN and circuits, the styles that had box annotations in training.
Out of scope. Charts, plots and tables.

License

Apache-2.0, like the base model. The CC BY 4.0 training sets are credited above.