What changed from v2
The first PageAudit was trained on pages we wrote ourselves. On real websites it lost clearly to Claude Sonnet 5.5. We studied why, then fixed three things:
- It reported some false contrast problems, for example text on top of photos. These cases are now filtered out.
- It did not check keyboard focus outlines that blend into the background. A new check covers this.
- Its strictness was tuned on our own pages and was too high for real sites. It was retrained on real-site examples and retuned.
How it works
PageAudit is a small pipeline, not a single prompt. A headless browser measures the page. A set of fixed checks turns the measurements into a list of possible problems, each with its numbers. This model then decides which ones are real problems and which are acceptable design choices. The model cannot audit a page without that list. The code for the first two steps is in the pipeline folder of this repository.
Results on real websites
We collected 358 real websites (old and new, government, universities, small organisations, businesses). 209 were used for study and training. 149 were kept aside and never used for training or tuning, and all numbers below come from those 149. The strictness setting was fixed on separate sites before the 149 were looked at.
Both systems received the same measured page data. Claude Opus 5.5 judged every reported problem, without knowing which system reported it, as real, false, or a legitimate design choice. A problem counts as found if Opus judged it real. "Recall" is the share of all problems judged real (from either system) that a system found.
Table with columns: 149 unseen real sites, PageAudit v3, Claude Sonnet 5.5| 149 unseen real sites | PageAudit v3 | Claude Sonnet 5.5 |
|---|
| Overall score (F1) | 66% | 44% |
| Real problems found (recall) | 73% | 30% |
| Reports that were correct (precision) | 61% | 77% |
| Real problems per page | 6.1 | 2.6 |
| Reports per page | 10.1 | 3.4 |
| Sites where the judge preferred the report | 83 | 49 (17 ties) |
PageAudit v3 reports about three times as many problems, so a fair comparison also matters at similar report length. With the setting made stricter so it reports 3.7 problems per page, v3 was correct 84% of the time and found 3.1 real problems per page, against 77% and 2.6 for Claude Sonnet 5.5.
For reference, the earlier version scored 41% against 74% for Claude Sonnet 5.5 on the 209 study sites.
Limitations
- The judge (Claude Opus 5.5) is from the same model family as the comparison system, and the training labels also came from Claude Opus. The results are judgments by a model, not by human auditors.
- Pages were rendered on one computer in a desktop browser at 1280 pixels wide. Mobile layouts, logged-in pages and pages that load slowly were not tested.
- The default setting favours finding problems over precision: about two in five reports may be wrong or a design choice. Raise the threshold for a shorter, more precise report.
- Page data is cut to a fixed size on very large pages, so some problems on those pages can be missed.
- Only English-language sites were sampled.
Training
Starting from the earlier PageAudit adapter, trained on about 1,800 examples from real websites labelled by Claude Opus 5.5, mixed with the earlier synthetic and hand-authored examples. Base model: Qwen3.5-4B.
License
Apache 2.0.