wjbmattingly

comma-qwen-3.5-9b-full

README

License: apache-2.0

Results

Greedy decoding, 3072 max new tokens, 2048 visual tokens per page, batch 4. "Base" is the stock Qwen/Qwen3.5-9B, scored on the same pages in the same job — a base number from another job on another day is not a control.

Table with columns: metric, base, this model
metric	base	this model
CER (NFD)	0.7058	0.1168
CER (raw code points)	0.7003	0.1575
WER (NFD)	1.6464	0.3541
line recall (NFD)	0.0070	0.1751
CER macro (NFD)	0.6904	0.0985
degenerate pages	0.1235	0.0000
truncated pages	0.2140	0.0041

CER/WER are micro-averaged (total edits ÷ total reference characters), so a long page outweighs a short one; the macro mean is given alongside because a gap between them says the errors are concentrated. line_recall is the share of reference lines reproduced exactly and in order — for a line-by-line CATMuS target that is the number a palaeographer looks at first.

Usage

python
import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

REPO = "wjbmattingly/comma-qwen-3.5-9b-full"
BASE = "Qwen/Qwen3.5-9B"

processor = AutoProcessor.from_pretrained(REPO)
model = AutoModelForImageTextToText.from_pretrained(
    REPO, dtype=torch.bfloat16, device_map="cuda"
)
model.eval()

# The prompt is the 11.6k-character CATMuS rule set the model was trained under.
# Serving a different prompt serves a different task: the rules are what the
# target obeys, so the model was taught to read them as part of the input.
prompt = open("prompt.txt", encoding="utf-8").read()

image = Image.open("page.jpg").convert("RGB")
messages = [{
    "role": "user",
    "content": [{"type": "image"}, {"type": "text", "text": prompt}],
}]

# `enable_thinking=False` is not optional. With no kwarg, Qwen3.5 0.8B/2B leave
# the thinking block CLOSED and 4B/9B leave it OPEN -- and a model trained on
# pure transcription that is handed an open block will think instead of
# transcribing, emitting prose until the token budget runs out.
text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
)
assert text.count("<think>") == text.count("</think>"), "thinking block left open"

inputs = processor(text=[text], images=[[image]], return_tensors="pt").to(model.device)

with torch.inference_mode():
    out = model.generate(
        **inputs,
        do_sample=False,             # greedy is the published control
        repetition_penalty=1.1,      # the measured serving default; 1.0 reproduces the table
        max_new_tokens=3072,
        eos_token_id=[processor.tokenizer.eos_token_id],
        pad_token_id=processor.tokenizer.pad_token_id,
    )

print(processor.tokenizer.decode(out[0][inputs["input_ids"].shape[1]:],
                                 skip_special_tokens=True).strip())

prompt.txt is in the demo Space.

The image budget travels with the checkpoint

This repo's processor_config.json is already capped to 2048 visual tokens per page (2,097,152 pixels after smart-resize, one token per 32×32 block), which is what training and scoring used. Load the processor from this repo rather than from the base model: the library default is 16,777,216 pixels — 16k visual tokens for one page — and serving at that budget shows the model a page at a resolution it never saw.

Training

Table

tuning	full fine-tune
trainable parameters	9,409,813,744 of 9,409,813,744 (100.0%)
epochs	3
effective batch	8
learning rate	5e-05
max sequence length	8192
visual tokens per page	2048
precision	bf16
hardware	NVIDIA H200

max_length never truncates: every sample is measured first and over-budget ones are dropped, because a truncated completion teaches the model to stop mid-transcription and a truncated prompt removes the rules the target obeys.

Limitations

Nothing under 0.02 CER is a result. Four seeds of one configuration in this project gave CER 0.1271 / 0.1334 / 0.1370 / 0.1485 — mean 0.1365, sd 0.0090. bf16 training is nondeterministic across nodes and greedy decoding amplifies a sub-millivolt logit difference into a different token. Treat differences smaller than ~0.02 as ties.
Greedy is deterministic given identical batching, but not batch-size invariant. Re-scoring the same pages at batch 1 instead of 4 changes almost every prediction and moves aggregate CER by ±0.013 for the smallest model here. Any comparison across a batch-size change is invalid.
Quote cer_nfd, not cer. ũ (U+0169) and u + combining tilde are one glyph, and 77% of training targets mix the two forms — so an edit distance over raw code points charges two edits for a difference that is not on the page, about 16% of all edit operations. Every checkpoint is biased toward one form and training does not fix it (the targets are mixed, so there is no signal to fit), which means raw cer partly scores which Unicode form a tokenizer prefers. NFD and not NFC: t̃ r̃ m̃ p̃ c̃ q̃ have no precomposed form.
The remaining error is mostly convention, not reading. On the best run of this project, word-division spaces are ~17% of all edit operations and allographs another ~5%, against ~1% for genuine letter confusion (r↔, ↔). Both are decisions about the transcription convention. Do not read a CER at this level as a statement about how well the model reads the script.

Citation

bibtex
@misc{comma_qwen35,
  title  = {comma-qwen-3.5: CATMuS transcription models for medieval Latin manuscripts},
  author = {Mattingly, William J. B.},
  year   = {2026},
  url    = {https://huggingface.co/wjbmattingly/comma-qwen-3.5-9b-full}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

wjbmattingly

Model Tree

Base

Qwen/Qwen3.5-9B

Fine-tuned

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality