GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
wjbmattingly
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
wjbmattingly
Model Tree
Qwen/Qwen3.5-0.8B
Input Modalities
Output Modalities
Supported Functionality
Qwen/Qwen3.5-0.8B — the stock vision-language model.comma-qwen-3.5-0.8b-full — 9,906 pages of medieval Latin under the CATMuS
graphemic standard (cer_nfd 0.1458), then continued to 33,112 pages
(cer_nfd 0.1240).The medieval ancestry is not incidental. Before NARA training the model applied
CATMuS conventions to English: asked to read a pension form it wrote SERUICE
for SERVICE, because rule B5 of its previous prompt maps v to u. That is
what the NARA fine-tuning had to undo.
100 held-out pages from 4 documents that appear nowhere in training. The full holdout is 762 pages across 39 documents; a document is held out whole, because the corpus averages ~27 pages per document in the same hand on the same printed form, so splitting by page would measure memorisation rather than reading.
| metric | value |
|---|---|
| CER | 0.5632 |
| CER (NFD) | 0.5632 |
| WER | 0.5141 |
| line recall | 0.1462 |
| truncated pages | 4.0% |
| degenerate pages | 2.0% |
Greedy decoding, 3072 max new tokens, 2048 visual tokens per page, batch 4.
Read the evaluation narrowly. It covers 100 pages taken in reading order from the front of the holdout, which turns out to span only 4 of the 39 held-out documents — roughly four hands and four printed forms. It is a spot check, not a characterisation of the model across the archive.
This is also an intermediate checkpoint from a run that is still going: epoch 1 of 3 on the 40,000-row set, published because step-based checkpointing made it available, not because training converged. Expect a later checkpoint to supersede it.
On those same 100 pages the checkpoint does beat its own parent: the 10,000-row model scores CER 0.6723 against this one's 0.5632, with line recall rising 0.1222 to 0.1462. Both were scored with the same code on the same pages.
Error on this material is not evenly spread. A typical page of connected prose scores far better than the headline; the error concentrates on sparse catalogue cards, where a model that has not learned to stop transcribes the printed grid rules until it hits the token ceiling. One such page — 43 reference characters against 6,141 generated — can contribute as much edit distance as a hundred ordinary pages, because CER is micro-averaged over characters.
Two things reduce it: more training data (going from 1,000 to 10,000 pages cut
truncation from 15% to 2% on the 2B), and repetition_penalty=1.1 at inference,
which roughly halved CER on an earlier checkpoint. Use the penalty when serving.
python
import torchfrom PIL import Imagefrom transformers import AutoModelForImageTextToText, AutoProcessorREPO = "wjbmattingly/nara-qwen-3.5-0.8b-ep1"model = AutoModelForImageTextToText.from_pretrained(REPO, dtype=torch.bfloat16,device_map="auto")processor = AutoProcessor.from_pretrained(REPO)# The prompt is part of the model. It ships in this repo as prompt.txt and must# be used verbatim: the model was trained to answer this exact instruction, and# a paraphrase is a different task than the one it was fine-tuned on.from huggingface_hub import hf_hub_downloadprompt = open(hf_hub_download(REPO, "prompt.txt")).read()image = Image.open("page.jpg").convert("RGB")messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": prompt}]}]text = processor.apply_chat_template(messages, tokenize=False,add_generation_prompt=True,enable_thinking=False)inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=3072, do_sample=False,repetition_penalty=1.1)print(processor.tokenizer.decode(out[0][inputs["input_ids"].shape[1]:],skip_special_tokens=True))
enable_thinking=False is required, not optional. Qwen3.5's chat template leaves
the thinking block open by default at 4B and above; a model trained on a
closed block will then reason instead of transcribing and never reach the text.
wjbmattingly/si-test —
38,756 pages from 1,407 documents, the first 40,000 rows of the
dataset. Targets are line-by-line transcriptions; --- marks a horizontal rule.