Evaluation
Table with columns: metric, value| metric | value |
|---|
| CER | 4.78 % |
| WER | 18.19 % |
| samples scored | 200 |
| characters scored | 6772 |
| character errors | 324 |
Measured on this run's own held-out validation split (page-level and seeded, so no page contributes lines to both sides). It is not a score on a shared benchmark and does not transfer to a different corpus.
The score mixes two kinds of validation, and the difference matters. dh-unibe/image-text_kurrent-xix held whole projects out of training, so those lines test unseen hands. dh-unibe/image-text_zh-regierungsratsprotokolle, dh-unibe/image-text_parlamentsdienste-protokolle, dh-unibe/image-text_nr-sr-vereinigte-bundesversammlung-xix contributed a seeded partition of their own training projects instead — unseen pages in a hand the model trained on, which is the easier test. The figure above is one CER over both, so read it as mostly in-domain, not as a held-out-hands benchmark. Scoring the held-out projects on their own would give the stricter number.
Input granularity: give it one line, not a page
Trained on line crops (granularity: line, 262 144 pixels), and its 6.80 %
is a line-level figure. Asked for a whole page it returns a fragment.
Measured 2026-09-21 on 27 pages of Lassberg correspondence — a corpus without
ground truth, so this is a description rather than a CER: 335 characters a page
against dh-unibe/trocr-kurrent's 844 on the same pages, readings like "1841",
"den 14. April 1849." or "der Böhne, der von der Hrn. Prof. von Hrn. Prof.", and
one page that degenerated into "1000000000…" for the whole 4096-token ceiling.
Two sibling models measured against ground truth show the same pattern with
numbers: dh-unibe/qwen3vl-german-xix-v2 reads whole pages at CER 0.98 (lines
0.052), dh-unibe/qwen3vl-medieval-german-v3 at 1.00 (lines 0.111). Segment
the page first and send one crop per call.
Details: thodel/serving-atr-inference#165.
Training data
dh-unibe/image-text_zh-regierungsratsprotokolle
- Training projects: —
- Evaluation: a seeded page-level split of the training projects (
partition=0.9, seed=42)
- Page cap: 20000
dh-unibe/image-text_parlamentsdienste-protokolle
- Training projects:
1849_01, 1849_02, 1849_03, 1849_04, 1849_05, 1849_06, 1849_07, 1849_08, 637157213248534270_verkleinert
- Evaluation: a seeded page-level split of the training projects (
partition=0.9, seed=42)
- Page cap: 138
dh-unibe/image-text_nr-sr-vereinigte-bundesversammlung-xix
- Training projects:
Nationalrat_01__Sitzung_–_06_11_1848, Nationalrat_02__Sitzung_–_07_11_1848, Nationalrat_03__Sitzung_–_08_11_1848, Nationalrat_04__Sitzung_–_09_11_1848, Nationalrat_05__Sitzung_–_10_11_1848, Nationalrat_06__Sitzung_–_13_11_1848, Nationalrat_07__Sitzung_–_14_11_1848, Nationalrat_08__Sitzung_–_15_11_1848, Nationalrat_09__Sitzung_–_18_11_1848, Nationalrat_10__Sitzung_–_21_11_1848, Nationalrat_11__Sitzung_–_22_11_1848, Nationalrat_12__Sitzung_–_23_11_1848, Nationalrat_13__Sitzung_–_24_11_1848, Nationalrat_14__Sitzung_–_25_11_1848, Nationalrat_15__Sitzung_–_27_11_1848, Nationalrat_16__Sitzung_–_28_11_1848, Nationalrat_17__Sitzung_–_29_11_1848, Ständerat_01__Sitzung_–_06_11_1848, Ständerat_02__Sitzung_–_08_11_1848, Ständerat_03__Sitzung_–_09_11_1848, Ständerat_04__Sitzung_–_11_11_1848, Ständerat_05__Sitzung_–_14_11_1848, Ständerat_06__Sitzung_–_15_11_1848, Ständerat_07__Sitzung_–_21_11_1848, Ständerat_08__Sitzung_–_23_11_1848, Ständerat_09__Sitzung_–_24_11_1848, Ständerat_10__Sitzung_–_25_11_1848, Ständerat_11__Sitzung_–_27_11_1848, Ständerat_12__Sitzung_–_28_11_1848, Ständerat_13__Sitzung_–_28_11_1848, VBV_01__Sitzung_–_16_11_1848, VBV_02__Sitzung_–_17_11_1848, VBV_03__Sitzung_–_20_11_1848, VBV_04__Sitzung_–_24_11_1848, VBV_05__Sitzung_–_29_11_1848
- Evaluation: a seeded page-level split of the training projects (
partition=0.9, seed=42)
- Page cap: 52
dh-unibe/image-text_kurrent-xix
- Training projects:
TRAIN_CITlab_Bassermann_0_4, TRAIN_CITlab_Bassermann_Manuscripts, TRAIN_CITlab_Bassermann_Manuscripts_0_2, TRAIN_CITlab_Binder_Kochbuch_2, TRAIN_CITlab_Escher_M1, TRAIN_CITlab_Gusbeth, TRAIN_CITlab_Handschriftliche_Archivquellen, TRAIN_CITlab_Konzilsprotkolle_B_Schwartz_5_2018, TRAIN_CITlab_MargareteSick, TRAIN_CITlab_MargareteSick20180731, TRAIN_CITlab_MargareteSick20180731a, TRAIN_CITlab_MÜLLER, TRAIN_CITlab_Protokoll_Hoftheater_1806, TRAIN_CITlab_Rehlen_1834_a, TRAIN_CITlab_RvE_Barlaam_HS_D_, TRAIN_CITlab_Steiner, TRAIN_CITlab_Suppes, TRAIN_CITlab_Suppes_3500, TRAIN_CITlab_Tagebuch_Arnold_v1, TRAIN_CITlab_umkc_Roland_M1, TRAIN_CITlab_umkc_Roland_M2, hufeland_privatbesitz_1829, nn_msgermqu2124_1827, nn_msgermqu2345_1827, parthey
- Evaluation: held-out projects
TEST_CITlab_Bassermann_0_4, TEST_CITlab_Bassermann_Manuscripts, TEST_CITlab_Bassermann_Manuscripts_0_2, TEST_CITlab_Binder_Kochbuch_2, TEST_CITlab_Escher_M1, TEST_CITlab_Gusbeth, TEST_CITlab_Handschriftliche_Archivquellen, TEST_CITlab_Konzilsprotkolle_B_Schwartz_5_2018, TEST_CITlab_MargareteSick, TEST_CITlab_MargareteSick20180731, TEST_CITlab_MargareteSick20180731a, TEST_CITlab_MÜLLER, TEST_CITlab_Protokoll_Hoftheater_1806, TEST_CITlab_Rehlen_1834_a, TEST_CITlab_RvE_Barlaam_HS_D_, TEST_CITlab_Steiner, TEST_CITlab_Suppes, TEST_CITlab_Suppes_3500, TEST_CITlab_Tagebuch_Arnold_v1, TEST_CITlab_umkc_Roland_M1, TEST_CITlab_umkc_Roland_M2
- Page cap: 19808
Materialized from that selection: 26,186 pages, 964,472 transcribed lines, 966,748 training samples.
Trained with the instruction: Transcribe the handwritten text in this image exactly as written. — serving it with different wording is a silent distribution shift.
Hyperparameters
granularity: lineprompt: Transcribe the handwritten text in this image exactly as written.load_in_4bit: falselora_r: 64lora_alpha: 128lora_dropout: 0.05target_modules:- q_proj- k_proj- v_proj- o_proj- gate_proj- up_proj- down_projmodules_to_save: []epochs: 1max_epochs: nullpatience: 2min_delta: 0.0001batch_size: 16accumulate_grad_batches: 1lrate: 0.0002lr_scheduler: cosinewarmup_ratio: 0.05weight_decay: 0.0max_grad_norm: 1.0optim: paged_adamw_8bitgradient_checkpointing: truesave_steps: 200max_pixels: 262144max_seq_len: 1024min_train_chars: 0eval_samples: 200max_new_tokens: nullseed: 42workers: 8device: cuda:0wandb_run: null
Provenance
Table | |
|---|
| engine | vllm |
| base model | Qwen/Qwen3.5-4B |
| training job | 20260918T070300Z-qwen3.5-4b-german-xix-v2 |
| code | evaluated with 85959f89227b |
| trained | 2026-09-19T00:19:44.747942+00:00 |
| weights | adapter_config.json, adapter_model.safetensors, chat_template.jinja, processor_config.json, tokenizer.json, tokenizer_config.json, training_summary.json |
metadata.json in this repo is the record the trainer wrote, verbatim: the full request, the parsed metrics and the job id.
Using it
This is a LoRA adapter, not a full model — it needs its base:
from peft import PeftModelfrom transformers import AutoModelForImageTextToText, AutoProcessor base = AutoModelForImageTextToText.from_pretrained('Qwen/Qwen3.5-4B')model = PeftModel.from_pretrained(base, 'dh-unibe/qwen3.5-4b-german-xix-v2')processor = AutoProcessor.from_pretrained('dh-unibe/qwen3.5-4b-german-xix-v2', trust_remote_code=True)
vLLM 0.11 will not serve it as an adapter (it refuses LoRA on the vision tower), so serving means merging it into the base first — scripts/merge_loras.py in serving-atr-inference does that.
Notes
HELD-OUT RESULT. On the published benchmark 'Handwritten Text Recognition Test Set: Minutes of the Swiss Federal Council (1848-1903)' (Hodel & Schoch 2021, Zenodo, https://doi.org/10.5281/zenodo.4746342; private HF mirror dh-unibe/image-text_federal-minutes-testset), all 2,751 lines, no document shared with training: CER 0.0680, WER 0.2342, length_ratio 0.9996.
WHAT THIS REPLACES. qwen3.5-4b-german-xix-v1 scored CER 0.3596 on the same lines and wrote little more than the first word on 807 of them (29.3%). v2 collapses on 0. The only difference from v1 is the corpus: v1 was built before the PageXML fix 33f55fc (#125), which had truncated 6.35x the characters of nr-sr-vereinigte-bundesversammlung-xix and 4.54x of parlamentsdienste-protokolle to their first word. Same four repositories, same seed, same page-level split, same hyperparameters.
THE CER IN THE TABLE ABOVE (0.0478) IS THE VALIDATION SPLIT, not the benchmark, and it is NOT comparable with v1's 0.0107: v1 was scored on the first 200 lines of val.jsonl (five in-domain pages), v2 on the stratified draw introduced in #120. Quote 0.0680 as this model's accuracy on unseen hands, and 0.0478 only as its score on its own corpus.
WHERE IT SITS. On the same 2,751 benchmark lines, the four models trained on the corrected corpus score: qwen3.5-4b-german-xix-v2 CER 0.0680, qwen3vl-german-xix-v2 CER 0.0765, qwen3.5-2b-german-xix-v2 CER 0.0895, qwen3.5-0.8b-german-xix-v2 CER 0.1115. None of the four collapses on a single line.
The remaining errors are 4,995 substitutions against 1,437 missing characters - misreadings rather than lost text.
Licence
This repo: mit — see LICENSE.
Base model Qwen/Qwen3.5-4B: apache-2.0 (per its hub card). These weights are a derivative of it.