docling-project

ChemicalOCR

README

License: apache-2.0

Overview

ChemicalOCR is fine-tuned from SmolDocling, a lightweight 256M-parameter model originally developed for end-to-end document conversion. By specializing for chemical images, ChemicalOCR substantially outperforms general-purpose OCR systems on chemical structure recognition tasks.

What it does

Extracts character-level text and bounding boxes from chemical structure images
Recognizes atom labels, chemical abbreviations, and descriptive text near or below molecular structures

Architecture

Base model: SmolDocling-256M (Idefics3ForConditionalGeneration)
Parameters: 256M
Input: Chemical structure image
Output: Text detections with bounding boxes

Performance

ChemicalOCR substantially outperforms existing OCR models on chemical structure benchmarks:

Table with columns: Benchmark, ChemicalOCR (Ours), PaddleOCR v5, EasyOCR
Benchmark	ChemicalOCR (Ours)	PaddleOCR v5	EasyOCR
M2S (103 images) — F1@IoU₀.₅	87.2	7.7	10.2
USPTO-M (74 images) — F1@IoU₀.₅	93.0	1.2	18.0
IP5-M (1000 images) — F1@IoU₀.₅	86.5	1.9	18.4

Detailed Metrics

Table with columns: Benchmark, Precision, Recall, F1, Accuracy@IoU₀.₅
Benchmark	Precision	Recall	F1	Accuracy@IoU₀.₅
M2S	86.9	87.4	87.2	32.0
USPTO-M	93.5	92.6	93.0	63.5
IP5-M	85.6	87.4	86.5	69.5

Usage

python
from transformers import AutoProcessor, AutoModelForVision2Seq
from PIL import Image

model = AutoModelForVision2Seq.from_pretrained("docling-project/ChemicalOCR")
processor = AutoProcessor.from_pretrained("docling-project/ChemicalOCR")

image = Image.open("chemical_structure.png")
inputs = processor(images=image, return_tensors="pt")
outputs = model.generate(**inputs)
result = processor.decode(outputs[0], skip_special_tokens=True)

Part of MarkushGrapher-2

ChemicalOCR is a key component of MarkushGrapher-2, an end-to-end model for multimodal recognition of chemical structures from patent documents. In the MarkushGrapher-2 pipeline, ChemicalOCR provides the text and layout information that is fused with visual features for Markush structure recognition.

Citation

bibtex
@inproceedings{strohmeyer2026markushgrapher2,
  title     = {MarkushGrapher-2: End-to-end Multimodal Recognition of Chemical Structures},
  author    = {Strohmeyer, Tim and Morin, Lucas and Meijer, Gerhard Ingmar and Weber, Valery and Nassar, Ahmed and Staar, Peter W. J.},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2026}
}

License

This model is released under the Apache 2.0 License.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

docling-project

Model Tree

Base

docling-project/SmolDocling-256M-preview

Fine-tuned

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality