OmniGnosis

Leibniz-HTR-HMER-Adapter

Model Details

Base Model: phxember/Uni-MuMER-Qwen3.5-2B
Fine-Tuning Method: 4-bit Quantized Low-Rank Adaptation (QLoRA) using the Unsloth library.
Trainable Parameters: 4.01% (92.4 million parameters out of 2.306 billion).
Target Dataset: Leibniz-HME (Spatial-Constraint split: 2,642 training, 968 validation, 961 test expressions).

Training Settings

LoRA Parameters: Rank $r = 64$ , Alpha $α = 128$ , Dropout $= 0$ .
Optimizer: Paged AdamW (8-bit)
Learning Rate: $1 0 - 5$ with a cosine decay scheduler and 3% warmup.
Batch Size: 4 (Gradient Accumulation: 4)
Max Steps: 12,000 steps
Input Canvas: Letterboxed to a $768 \times 768$ pixel canvas.

Evaluation Results (Spatial-Constraint Split)

Evaluated on the 961-sample S-C test split for HMER (LaTeX Expression Recognition Rate - ExpRate and token Edit Distance (ED)):

Table with columns: Task Strategy, ExpRate (%), Edit Distance (ED), ED ≤1 (%), ED ≤2 (%)
Task Strategy	ExpRate (%)	Edit Distance (ED)	ED $\leq 1$ (%)	ED $\leq 2$ (%)
Uni-MuMER Qwen3.5 2B (Public Baseline)	27.89	3.88	44.75	57.65
adapted QLoRA PLAIN	70.55	1.14	81.58	86.16

Key Findings

Historical Adaptation: Fine-tuning via QLoRA closes a huge domain shift (ExpRate jumps from 27.89% to over 73.15%).
Multi-task Auxiliary Objectives: Combining Symbol Counting (SC), Tree-CoT, and Error-Driven Learning (EDL) corrections yields the strongest overall top-1 accuracy (73.15% ExpRate).
Tree-CoT: Serializing the Abstract Syntax Tree (AST) structure during training provides the strongest performance on hard, complex expressions (39.18% ExpRate on hard items).

Prompt Formats

This adapter was trained on the following prompt formats, which can be tested using the accompanying full-stack application:

1. Plain

Prompt: "I have an image of a handwritten mathematical expression. Please write out the expression of the formula in the image using LaTeX format"

2. Symbol Counting

Prompt: "I have an image of a handwritten mathematical expression. Please identify and count each distinct visible mathematical symbol in the image, and then provide its corresponding LaTeX format"

3. Tree-CoT (AST Serialization)

Prompt: "I have an image of a handwritten mathematical expression. Please generate the abstract syntax tree (AST) of the formula in the image, and then provide its corresponding LaTeX format."

4. EDL Error Detection (DETECT)

Prompt: "I have an image of a handwritten mathematical expression and its OCR recognition result. Please help me to detect possible errors in the recognition result and mark the places where errors occur with <error_start> <error_end> and <deleted>.
erroneous formula: [candidate]
Marked formula:"

5. EDL Error Correction (CORRECT)

Prompt: "I have an image of a handwritten mathematical expression and a predicted formula with error tags, correct the formula by modifying the parts marked with <error_start> and <error_end> and inserting content where <deleted> are present. Output the modifications as REPLACE, INSERT, or DELETE operations.
Marked formula: [tagged candidate]
Correction log:"

Developed as part of the PHILIUMM project, dedicated to Gottfried Wilhelm Leibniz's unpublished mathematical manuscripts.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

OmniGnosis

Model Tree

Base

phxember/Uni-MuMER-Qwen3.5-2B

Adapter

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

Model Details

Base Model: phxember/Uni-MuMER-Qwen3.5-2B
Fine-Tuning Method: 4-bit Quantized Low-Rank Adaptation (QLoRA) using the Unsloth library.
Trainable Parameters: 4.01% (92.4 million parameters out of 2.306 billion).
Target Dataset: Leibniz-HME (Spatial-Constraint split: 2,642 training, 968 validation, 961 test expressions).

Training Settings

LoRA Parameters: Rank $r = 64$ , Alpha $α = 128$ , Dropout $= 0$ .
Optimizer: Paged AdamW (8-bit)
Learning Rate: $1 0 - 5$ with a cosine decay scheduler and 3% warmup.
Batch Size: 4 (Gradient Accumulation: 4)
Max Steps: 12,000 steps
Input Canvas: Letterboxed to a $768 \times 768$ pixel canvas.

Evaluation Results (Spatial-Constraint Split)

Evaluated on the 961-sample S-C test split for HMER (LaTeX Expression Recognition Rate - ExpRate and token Edit Distance (ED)):

Table with columns: Task Strategy, ExpRate (%), Edit Distance (ED), ED ≤1 (%), ED ≤2 (%)
Task Strategy	ExpRate (%)	Edit Distance (ED)	ED $\leq 1$ (%)	ED $\leq 2$ (%)
Uni-MuMER Qwen3.5 2B (Public Baseline)	27.89	3.88	44.75	57.65
adapted QLoRA PLAIN	70.55	1.14	81.58	86.16

Key Findings

Historical Adaptation: Fine-tuning via QLoRA closes a huge domain shift (ExpRate jumps from 27.89% to over 73.15%).
Multi-task Auxiliary Objectives: Combining Symbol Counting (SC), Tree-CoT, and Error-Driven Learning (EDL) corrections yields the strongest overall top-1 accuracy (73.15% ExpRate).
Tree-CoT: Serializing the Abstract Syntax Tree (AST) structure during training provides the strongest performance on hard, complex expressions (39.18% ExpRate on hard items).

Prompt Formats

This adapter was trained on the following prompt formats, which can be tested using the accompanying full-stack application:

1. Plain

Prompt: "I have an image of a handwritten mathematical expression. Please write out the expression of the formula in the image using LaTeX format"

2. Symbol Counting

Prompt: "I have an image of a handwritten mathematical expression. Please identify and count each distinct visible mathematical symbol in the image, and then provide its corresponding LaTeX format"

3. Tree-CoT (AST Serialization)

Prompt: "I have an image of a handwritten mathematical expression. Please generate the abstract syntax tree (AST) of the formula in the image, and then provide its corresponding LaTeX format."

4. EDL Error Detection (DETECT)

Prompt: "I have an image of a handwritten mathematical expression and its OCR recognition result. Please help me to detect possible errors in the recognition result and mark the places where errors occur with <error_start> <error_end> and <deleted>.
erroneous formula: [candidate]
Marked formula:"

5. EDL Error Correction (CORRECT)

Prompt: "I have an image of a handwritten mathematical expression and a predicted formula with error tags, correct the formula by modifying the parts marked with <error_start> and <error_end> and inserting content where <deleted> are present. Output the modifications as REPLACE, INSERT, or DELETE operations.
Marked formula: [tagged candidate]
Correction log:"

Developed as part of the PHILIUMM project, dedicated to Gottfried Wilhelm Leibniz's unpublished mathematical manuscripts.

Leibniz-HTR-HMER-Adapter

README

Model Details

Training Settings

Evaluation Results (Spatial-Constraint Split)

Key Findings

Prompt Formats

1. Plain

2. Symbol Counting

3. Tree-CoT (AST Serialization)

4. EDL Error Detection (DETECT)

5. EDL Error Correction (CORRECT)

Explore FriendliAI today

README

Model Details

Training Settings

Evaluation Results (Spatial-Constraint Split)

Key Findings

Prompt Formats

1. Plain

2. Symbol Counting

3. Tree-CoT (AST Serialization)

4. EDL Error Detection (DETECT)

5. EDL Error Correction (CORRECT)