Peter-A-P

legible-2b-nist-v1120-s0-a72f03d21f

README

License: apache-2.0

What it reads and what it writes

Input: the document's pages as images, in page order, then the instruction every reader in the project is sent (legible.prompt, version 1), which describes each field and gives the JSON Schema. The adapter was trained on that instruction and nothing else.

One JSON object a page, for whichever of the twenty 1988 form faces it is: 20 schemas, 1,113 text fields in all, every value a string or null. The face's own prompt names it and lists its fields; the fields are the NIST tables' (legible/schema/nist_fields.json), check marks and signatures left out.

Table with columns: Face, Fields
Face	Fields
1040_1	73
1040_2	46
2106_1	28
2106_2	45
2441	12
4562_1	84
4562_2	98
6251	50
sch_a	46
sch_b	64
sch_c_1	51
sch_c_2	8
sch_d_1	191
sch_d_2	27
sch_e_1	95
sch_e_2	86
sch_f_1	62
sch_f_2	19
sch_se_1	9
sch_se_2	19

How it was trained

Project 06's QLoRA recipe with a vision encoder (legible/train/vision.py): one user turn of page images then the instruction, one assistant turn of the truth's JSON, the loss on the answer's tokens only. Every training document is from the split's train part; a held-out document refuses the set (legible/train/dataset.py).

Table with columns: Setting, Value
Setting	Value
Base	`google/gemma-4-E2B` at revision `d29ff6b45f081a49ee2733a859c9c9c2d95d1a6f`
Chat template	`google/gemma-4-E2B-it` at revision `3e22461f65e89153144f8adb70e3b8c2cc9845a7`
Quantisation	NF4 4-bit with double quantisation, bf16 compute; the vision and audio towers, the vision projector and the output layer kept at full precision
LoRA	rank 64, alpha 128, dropout 0.05, on the language model's attention and MLP projections and the vision projector, never the towers
Optimisation	learning rate 0.0001, cosine schedule, warmup 0.03, 1 epoch, 1 document a step, 8 steps accumulated
Visual tokens a page	1,120
Sequence limit	16,384 tokens
Seed	0
Training set	`nist` (3,850 documents; the first 800 taken)
Training documents and pages	800 documents, 800 pages
Optimiser steps	97, from 775 documents at 8 a step; 25 of them Schedule D page 1, over the sequence limit and left out by the run (legible/train/recipe.py)
Loss and timing	in `run.json`, the training record uploaded with the adapter; it was not on the machine that wrote this card
Faces trained on	1040_1 141, 1040_2 116, 2106_1 13, 2106_2 8, 2441 12, 4562_1 40, 4562_2 45, 6251 13, sch_a 56, sch_b 86, sch_c_1 29, sch_c_2 16, sch_d_1 25, sch_d_2 37, sch_e_1 48, sch_e_2 54, sch_f_1 16, sch_f_2 14, sch_se_1 14, sch_se_2 17

Training data and its licences

NIST Special Database 2 (1988 IRS tax forms, synthesised), https://www.nist.gov/srd/nist-special-database-2. Terms: Public domain in the United States (title 17 United States Code Section 105), confirmed by NIST to Peter Parker on 2026-10-17, including that adapters trained on the pages may be published.

"This data/work was created by employees of the National Institute of Standards and Technology (NIST), an agency of the Federal Government. Pursuant to title 17 United States Code Section 105, works of NIST employees are not subject to copyright protection in the United States. This data/work may be subject to foreign copyright. The data/work is provided by NIST as a public service and is expressly provided "AS IS." ..."

Quoted from https://www.nist.gov/open/license, the regime NIST confirmed applies to SD2. Attribution: NIST Special Database 2, DOI 10.18434/M3J08M.

The adapter is released under apache-2.0, the base model's licence. Publication: NIST SD2 is public domain in the United States; NIST confirmed on 2026-10-17 that adapters trained on it may be published.

Results

On the 500 NIST test faces every reader reads, read once, level 0, temperature 0, unconstrained. Field accuracy is VRDU's matchers, one grader for every reader; intervals are 95% bootstrap over documents. The adapter is costed at an RTX 3090 at US$0.50 an hour on RunPod (2026-10-18) over the measured wall time of its read; the frontier models at the gateway ledger's cost at the 2026-10-16 price list, batch rate.

Table with columns: Reader, Fields correct (95% CI), Every field right, Invented fields, Answers that did not parse, Cost per 1,000 pages, Documents
Reader	Fields correct (95% CI)	Every field right	Invented fields	Answers that did not parse	Cost per 1,000 pages	Documents
B, Gemma 4 E2B fine-tuned (this adapter)	73.6% (70.4 to 76.7)	27.6% (23.8 to 31.6)	1.4% (1.0 to 1.8)	4.8% (3.0 to 6.8)	US$0.40 (0.38 to 0.42)	500
C, Gemma 4 E2B untuned on the image	15.0% (13.7 to 16.4)	0.0% (0.0 to 0.0)	27.9% (25.3 to 30.4)	9.6% (7.2 to 12.2)	US$0.41 (0.39 to 0.43)	500
A, Claude Sonnet 5.5 on the image	86.7% (84.8 to 88.4)	46.2% (42.0 to 50.4)	0.7% (0.6 to 1.0)	0.0% (0.0 to 0.0)	US$16.38 (15.63 to 17.19)	500
A, Claude Haiku 5.5 on the image	84.5% (82.3 to 86.4)	36.8% (32.6 to 41.0)	0.5% (0.3 to 0.7)	0.4% (0.0 to 1.0)	US$1.12 (1.08 to 1.17)	500
A, GPT-6.1 Sol on the image	94.9% (94.0 to 95.6)	50.2% (45.8 to 54.6)	0.4% (0.3 to 0.5)	0.0% (0.0 to 0.0)	US$15.33 (14.69 to 16.00)	500
A, Gemini 3.8 Flash on the image	80.1% (76.2 to 83.7)	46.2% (41.8 to 50.6)	0.6% (0.4 to 0.8)	10.6% (8.0 to 13.4)	US$12.48 (11.85 to 13.16)	500

The best frontier model on these documents is GPT-6.1 Sol, at 94.9% (94.0 to 95.6) of fields for US $15.33 (14.69 t o 16.00) a t h o u s an d p a g es . T hi s a d a pt e r' s f i e l d a cc u r a cy d i f f er s f r o mi t b y - 21.2 p o in t s, a t U S$ 0.40 (0.38 to 0.42) a thousand pages, about 1/39 of the cost. Against its untuned base it reads +58.6 points.

Training variance: the same recipe at seeds 1 and 2, each on its own split's test set: seed 1, 74.0% (71.0 to 77.0); seed 2, 73.7% (70.1 to 77.2). The three seeds' point estimates span 0.4 points (their test sets differ where the split files do).

The 03 release gate

Not gated. Project 03's gate compared the E4B, the better of the two sizes on dev, for this corpus (docs/gate.md); its decision is on that card.

Failure modes

A form design it has not seen. Not measured on this corpus: NIST's twenty faces are fixed, and every face it reads it saw in training. On FARA and DeepForm, where it was measured, the fine-tunes read worse on templates they did not see and the frontier models do not (the README's template gap tables); expect worse on a form unlike its training set.
A bad scan. On NIST's 300 curve pages, degraded in the project's five reproducible steps (docs/degradation.md): level 0 74.9% (71.3 to 78.3), level 1 75.4% (72.0 to 78.8), level 2 73.9% (69.9 to 77.8), level 3 72.1% (68.4 to 75.6), level 4 68.7% (64.9 to 72.4). Beside it: GPT-6.1 Sol level 0 94.9% (93.8 to 95.8), level 1 95.1% (93.7 to 96.2), level 2 94.7% (93.3 to 95.7), level 3 74.6% (68.3 to 80.7), 239 of 300 read so far.
Confidently wrong. Its own token probability is not a reason to skip review. Of its dev fields written at a token probability of 0.999 or more (5,569), 85.0% (82.1 to 87.7) are right; the commonest error there is a null where the form carries a value (638). At 99% accuracy it sends 100.0% (100.0 to 100.0) of its test fields to a person, since no threshold reached 99% on dev. The probability measures how sure the model is of its habit, not whether the page agrees (docs/review.md).
Training variance. The same recipe at three seeds reads from 73.6% to 74.0% of fields on test (above); a retrained copy is not guaranteed to read like this one.
Not non-inferior to the frontier. Project 03's gate cannot decide for the E4B, the size it compared, on this corpus at a 3-point margin per field (above). Do not take it as a drop-in for a frontier model's read.
Malformed answers. 4.8% (3.0 to 6.8) of its test answers did not parse as JSON. It is read freely: holding the fine-tunes to the schema's grammar was tried on dev and rejected, because it pushed them off their trained spacing and the DeepForm E2B then answered null for every field (docs/rejected.md). A caller must count a parse failure as a document to type by hand.
Invented fields. 1.4% (1.0 to 1.8) of scored fields carry a value where the form's box is blank.
One instruction. Trained on one prompt and one schema; another wording, another field list or another language is untested.

Intended use

Reading NIST tax forms of the kind it was trained on into JSON for a person to check, at a fraction of a frontier model's cost per page; and reproducing the measurements in this card. Plan on a person checking every field: no confidence measured in the project separates right fields from wrong ones well enough to skip review at 99% (the README's honest limitation).

Out of scope

Any decision about a person made on its output without a person checking it.
Handwriting, other form families, long multi-page documents, or a form whose type is unknown.
Unattended use on form designs it was not trained on, where it is worst.
Pages that are not public: it was measured on public records and datasets only.

Loading it

python

from peft import PeftModelfrom transformers import AutoModelForImageTextToText, AutoProcessor base = AutoModelForImageTextToText.from_pretrained("google/gemma-4-E2B", revision="d29ff6b45f081a49ee2733a859c9c9c2d95d1a6f")model = PeftModel.from_pretrained(base, "<this repository>")processor = AutoProcessor.from_pretrained("<this repository>")processor.image_processor.max_soft_tokens = 1120

Read it at temperature 0, with the prompt legible.prompt.build_prompt writes for its schema; the project served it merged into the bf16 base under vLLM (legible serve merge).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Peter-A-P

Model Tree

Base

google/gemma-4-E2B

Adapter

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality