Training
- Pairs: from every GT-Lejonet training line (February 2026, the Swedish Lion's training side):
- the line's crop with its GT text (a match);
- about as many wrong candidates (rule perturbations of the GT text, and the line of the same page closest to it in
length);
- diffu-minicpm-ocr-htr's readings of 100,000 training lines, a match where the
reading equals the GT after normalisation and a mismatch where it does not.
- Rendering: each candidate is rendered by diffu's renderer (
htr-train-jev-render-1+ae7b7855e115+pillow-12.2.0+freetype-2.14.3). Scores depend on the renderer, so render
with the same one.
- Recipe: OpenBMB's (full fine-tune of the language model, the vision encoder and aligner frozen, learning rate
1e-5, ZeRO-3), 2 epochs, 128 pairs a step, unpacked, on two RTX PRO 6000 Blackwell GPUs.
- Checkpoint: checkpoint-45486, the last.
- Curves: the training and evaluation metrics are in this repo's
runs/ folder (TensorBoard, the Training metrics
tab).
Evaluation
On its held-out dev pairs (from training lines) at the last evaluation: accuracy 95.6%, loss 0.1315 (step 45486/45486).
On 2,000 pairs drawn from the test lines' pairs (786 correct transcriptions, 395 wrong readings by
the reader and 818 rule-made wrong candidates), scored in vLLM as below:
Table with columns: AUROC, At a threshold of 0.5 | AUROC | At a threshold of 0.5 |
|---|
| All pairs | 0.870 | 77.8% judged right |
| The correct text against rule-made wrong candidates | 0.988 | 96.9% of the correct kept, 93.9% of the wrong caught |
| The correct text against the reader's own wrong readings | 0.624 | 6.6% of the wrong caught |
OpenBMB's checkpoint before this fine-tuning, given the same prompt and rendering (not the ones it was trained with),
scores an AUROC of 0.594 on these pairs.
It does not yet catch the reader's own mistakes. They are small (a median CER of 5%, a letter or two in a
line) and were 1.2 % of its training pairs (35,578 of 3.0 million). A threshold of 0.97 catches 74% of them
but rejects 55% of the correct texts. Use it for gross mismatches (another line's text, dropped or swapped words);
for the reader's near-misses it needs many more real misreadings to train on.
This has not yet been measured on JevOCRBench, nor as a label filter on machine-transcribed lines.
Use
Score a line with the inputs made exactly as in training, or the scores shift:
- the crop: the line's bounding box, white outside its polygon, 48 pixels high;
- the prompt: diffu's own, in English (
diffu_htr_train.jev.prompt.model_prompt), not OpenBMB's;
- the rendering: diffu's renderer
htr-train-jev-render-1+ae7b7855e115+pillow-12.2.0+freetype-2.14.3 (diffu_htr_train.jev.render.render_text, DejaVu Sans, drawn with the
Pillow that string names).
So OpenBMB's Jev services, which render and prompt their own way, do not fit this model. Set up vLLM (a nightly from
29 October 2026 or later, with transformers 5.22.1) and the two functions:
uv venv --python 3.12 jev && source jev/bin/activateecho "transformers==5.22.1" > overrides.txtuv pip install --override overrides.txt "pillow==12.2.0" \ "vllm @ https://wheels.vllm.ai/fa3b9d1377219247060d3c8b49eae199c9e7e154/vllm-0.33.0rc1.dev234%2Bgfa3b9d137-cp38-abi3-manylinux_2_28_x86_64.whl" \ "git+https://github.com/Borg93/diffu2#subdirectory=packages/htr_train"
Then score a line:
import mathfrom pathlib import Path from PIL import Imagefrom vllm import LLM, PoolingParams from diffu_htr_train.jev.prompt import model_promptfrom diffu_htr_train.jev.render import render_text if __name__ == "__main__": jev = LLM( model="Gabriel/diffu-minicpm-ocr-htr-jev", runner="pooling", convert="classify", hf_overrides={"num_labels": 2}, limit_mm_per_prompt={"image": 2}, ) crop = Image.open("line.jpg").convert("RGB") text = "Anno 1649 den 6 Aprilis" rendering = render_text(text, Path("/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf"), crop.size) [out] = jev.classify( [{"prompt": model_prompt(text), "multi_modal_data": {"image": [crop, rendering]}}], pooling_params=PoolingParams(use_activation=False), ) no, yes = out.outputs.probs print(1 / (1 + math.exp(no - yes))) # the probability that the text matches the line
Without vLLM, transformers gives the same scores (within 0.01 on 60 test pairs):
import torchfrom transformers import AutoProcessor, Qwen3_5ForSequenceClassification processor = AutoProcessor.from_pretrained("Gabriel/diffu-minicpm-ocr-htr-jev")jev = Qwen3_5ForSequenceClassification.from_pretrained("Gabriel/diffu-minicpm-ocr-htr-jev", num_labels=2, dtype=torch.bfloat16).to("cuda")inputs = processor(text=[model_prompt(text)], images=[crop, rendering], return_tensors="pt").to("cuda")with torch.inference_mode(): print(float(torch.softmax(jev(**inputs).logits[0].float(), -1)[1]))
The score is not calibrated: choose a threshold on labelled lines of your own.
Licence
Apache-2.0, as its base model.