zzha6204

CheMM-R1

README

License: apache-2.0

Capabilities

CheMM-R1 is designed for the following tasks:

SmilesQA — predict the SMILES string of a molecule from its 2D structure image.
IupacQA — predict the IUPAC name of a molecule from its 2D structure image.
MwQA — predict the molecular weight of a molecule from its 2D structure image.
SpectraQA — predict the SMILES of a molecule from its spectral images (IR, ¹H-NMR, ¹³C-NMR, positive-ion MS, negative-ion MS).

The model produces explicit step-by-step reasoning in <think> tags, intermediate SMILES in <smiles> tags, and the final answer in <answer> tags.

Method

Base model

Architecture: Qwen2_5_VLForConditionalGeneration (Qwen2.5-VL-3B-Instruct)
Precision: bfloat16
Context length: up to 128K tokens, trained with max sequence length 20,480 (cold-start) and output length 5,120 (RL)

Training pipeline (CheMMGRPO)

Cold start (SFT) — the base model is fine-tuned on 40,000 chemistry QA instances from CheMM-Bench with multimodal Chain-of-Thought reasoning distilled from Gemini-2.5-Pro, to inject organic chemistry knowledge and reasoning patterns.
Reinforcement learning (GRPO) — the cold-started model is further optimised with GRPO using four chemistry-specific reward functions:
- smiles_acc — chemical validity of generated SMILES
- atom_acc, func_group_acc — structural accuracy
- output_format — format compliance with <think>/<smiles>/<answer> tags
- answer_acc — factual correctness via Levenshtein-based fuzzy match for textual answers (SMILES, IUPAC) and ±0.05 g/mol tolerance for molecular weights

Key hyperparameters

Framework: MS-SWIFT
Cold start: lr 1e-4, Adam (β1=0.9, β2=0.95, ε=1e-8), batch size 64, max model length 20,480, max image pixels 262,144
RL (GRPO): 4 rollouts per question, sampling temperature 1.0, lr 1e-4, KL coefficient β=0.001, batch size 48, max output length 5,120
Hardware: 8× H200 (141 GB) GPUs, ~12 hours total training

Dataset: CheMM-Bench

CheMM-R1 is trained and evaluated on CheMM-Bench, a multimodal chemistry reasoning benchmark with 48,500 long Chain-of-Thought reasoning steps across four tasks:

26,500 molecules for structure recognition (SmilesQA / IupacQA / MwQA)
22,000 molecules for structure elucidation (SpectraQA) with five spectral image types (IR, ¹H-NMR, ¹³C-NMR, +ion MS, −ion MS)

Molecules are derived from the Alberts spectroscopic dataset (USPTO reaction database). SMILES → 2D structure conversion is performed with RDKit; IUPAC names are sourced from PubChem.

Results

Averaged accuracy / F1 on CheMM-Bench (CheMM-R1 vs. strongest baselines):

Table with columns: Model, SR Avg ACC, SR Avg F1, SpectraQA ACC, SpectraQA F1, Overall ACC, Overall F1
Model	SR Avg ACC	SR Avg F1	SpectraQA ACC	SpectraQA F1	Overall ACC	Overall F1
GPT-o3	5.78	10.94	1.50	2.96	3.34	6.46
Gemini-2.5-Pro	16.13	27.78	1.80	3.54	7.95

Tanimoto@1.0 structural match on SmilesQA / SpectraQA:

Table with columns: Model, SmilesQA, SpectraQA
Model	SmilesQA	SpectraQA
Gemini-2.5-Pro	44.52	11.46
Grok-4	2.18	18.28
ChemVLM-8B	31.99	—
CheMM-R1 (3B)	60.00	56.57

See the paper for full tables including BLEU-1 and Levenshtein-distance similarity and ablations on cold-start vs. GRPO vs. CheMMGRPO.

Usage

python
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
from PIL import Image
import torch

model_id = "zzha6204/CheMM-R1"

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

image = Image.open("molecule.png")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": "What is the SMILES representation of this molecule?"},
        ],
    }
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)

output_ids = model.generate(**inputs, max_new_tokens=2048)
response = processor.batch_decode(
    output_ids[:, inputs.input_ids.shape[1]:], skip_special_tokens=True
)[0]
print(response)

For SpectraQA, pass all available spectral images (IR, ¹H-NMR, ¹³C-NMR, +ion MS, −ion MS) as a multi-image message.

Output format

CheMM-R1 is trained to produce:

markdown
<think> step-by-step chemical reasoning </think>
<smiles> intermediate SMILES </smiles>
<answer> final answer </answer>

Downstream parsers should extract the content of <answer> as the final prediction.

Intended use and limitations

Intended for research on multimodal chemistry reasoning: molecular structure recognition and spectral elucidation of small organic molecules.
Molecules are drawn from the USPTO-derived Alberts dataset; out-of-distribution performance on larger natural products, organometallics, or experimentally noisy real-world spectra is not guaranteed.
Outputs — including SMILES, IUPAC names, molecular weights, and reasoning traces — may be incorrect and must not be used for safety-critical decisions in chemistry or medicinal research without expert verification.
The model is derived from Qwen2.5-VL-3B-Instruct and inherits its license and any biases of the base model and distilled reasoning data (Gemini-2.5-Pro).

Citation

If you use CheMM-R1 or CheMM-Bench in your research, please cite:

bibtex
@inproceedings{huang2025chemmr1,
  title     = {CheMM-R1: Enhancing Chemical Structure Recognition and Elucidation with Reasoning Multimodal Large Language Models},
  author    = {Huang, Liting and Zhang, Zhihao and Wang, Shoujin},
  year      = {2025}
}

Acknowledgements

Base model: Qwen2.5-VL-3B-Instruct
Training framework: MS-SWIFT
Spectroscopic data: Alberts et al. (USPTO-derived), PubChem, RDKit

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

zzha6204

Model Tree

Base

Qwen/Qwen2.5-VL-3B-Instruct

Fine-tuned

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer