BlazingCustoms

plateclerk-verdict-1.5b

Deploy Dedicated

README

License: apache-2.0

Scope — binding

Entertainment and exploration only.
This is NOT a content-moderation system. It was never built, tuned or measured as one.
It must not be used to make decisions about real people — not applications, not eligibility, not enforcement, not screening.
It reproduces California DMV reviewers' readings, including their biases and their inconsistencies. It was trained to imitate a particular set of human decisions, not to be correct, fair or consistent. Where those reviewers disagreed with each other or with themselves, the model learned the disagreement.
It emits offensive strings by design, because reading offensive intent in obfuscated text is the task.

Anything outside that is out of scope and unmeasured.

Headline metric

ROC-AUC = 0.7393 on 4,664 held-out rows, bf16 reference (transformers, greedy).

AUC is the headline, and accuracy deliberately is not. The classes are 80/20, so a model that answers DENIED to everything scores 80.30% accuracy. Accuracy alone once made a completely dead 3B arm — 0 APPROVED predictions on all 4,664 rows — read as "80.3%, just under the 0.5B", i.e. mild underperformance instead of a corpse. AUC is threshold-free, so it measures whether the model can rank approvals at all.

Table with columns: comparator, ROC-AUC, accuracy, balanced acc, APPROVED recall
comparator	ROC-AUC	accuracy	balanced acc	APPROVED recall
this model (tuned)	0.7393	82.55%	61.42%	26.55%
untuned `Qwen/Qwen2.5-1.5B-Instruct`	0.5515	52.38%	53.51%	55.39%
always-DENIED (majority)	0.5000	80.30%	50.00%	0.00%
random	0.5000	—	50.00%	—
shuffled-label AUC

Tuning moved AUC +0.1878 over this model's own untuned base.

That baseline was measured for this model, not inherited. The untuned figure above is Qwen/Qwen2.5-1.5B-Instruct scored on the same 4,664 rows through the same code path. Only the 0.5B's base AUC (0.5392) had ever been recorded; reusing it here would have attributed one model's number to another, since a different base starts from a different place.

For proportion: tuning is worth +0.1878 AUC on this model's own base, while tripling the parameters (0.5B -> 1.5B, both tuned) is worth +0.0219 [CI +0.0088, +0.0358]. Tuning is roughly 8x the lever that capacity is on this task. On accuracy alone the two sizes are indistinguishable (82.42% vs 82.53%, McNemar chi2 0.08, n.s.) — the gain is real and it is visible only in the ranking, which is why AUC is the headline.

Confusion matrix at the greedy operating point (gold ↓ / predicted →), from the f16 control:

Table with columns: APPROVED, DENIED
	APPROVED	DENIED
APPROVED	237	682
DENIED	130	3615

Greedy sits at the model's implicit 50% operating point, which on a 20/80 task is the wrong place to stand — hence the low APPROVED recall. A cross-fitted threshold (chosen on one half of the rows, scored on the other, split by plate hash so no plate straddles the halves) trades accuracy for balance: 66.10% accuracy at 66.16% balanced and 66.27% APPROVED recall. The threshold is never fitted on the rows it is reported on.

Quantisation — measured, not assumed

Every quant was evaluated on the same held-out rows with the same grader as the reference. Quantisation is not assumed lossless.

The comparison is decomposed on purpose. Comparing a GGUF quant directly against the bf16 reference confounds two changes — precision and serving path (transformers vs llama.cpp). So the array carries an f16 GGUF control: bf16→f16 isolates the serving path, f16→QN isolates quantisation. Only the second is reported as a quantisation cost. The f16 GGUF reproduces the bf16 reference to within 0.0004 AUC, so the serving path is clean and every delta below is genuinely quantisation.

Table with columns: variant, size, ROC-AUC, ΔAUC vs f16, 95% CI (paired), balanced acc, APPROVED recall, accuracy, shuffled-label floor
variant	size	ROC-AUC	ΔAUC vs f16	95% CI (paired)	balanced acc	APPROVED recall	accuracy	shuffled-label floor
f16	3.09 GB	0.7397	—	reference	61.16%	25.79%	82.59%	0.5036
Q8_0	1.65 GB

CIs are paired bootstraps over the same rows (10,000 resamples for token-F1, 2,000 for AUC). The floors are recomputed for every variant, never inherited.

Artifact digests

f16 — sha256:87010c95cbfac0a369b10d7f37f7f2417c65aeac4a61f9e27ead2ff85ce3b3e7 (3.09 GB)
Q8_0 — sha256:4b0459ccb6212707b260ff68cab019e7f866a0fca986389346871fd5ffdc94a5 (1.65 GB)
Q5_K_M — sha256:31b92f8be9c37b822e4977d550539382b23acf7b1f9dd36fce823deffc96c880 (1.13 GB)
Q4_K_M — sha256:3d7bb52ab3f7dffb78c051eec7e9713a267d535c5e3a6f3035503a47a7f67a2d (0.99 GB)

The oracle, and what it does NOT prove

Grading is done by a program, never by an LLM judge.

Grader: eval/plateclerk_verdict.py — sha256:1824e2e8acb97a25. It extracts the VERDICT: line and compares it to the gold label. That is all it does.
Mutation-tested against scripts/mutations/plateclerk-verdict-mutations.py (sha256:0341c49f6c9f2376): 4,514 mutants, 4,514 killed (100%), 0 false negatives.
Held-out split: eval/dmv-plateclerk-verdict-v2_test.jsonl, 4,664 rows, sha256:ef3c16ae5361062f.... The model never trained on these rows.

What it proves: the model's stated verdict matches the DMV's recorded decision on rows it has never seen, at a rate a program can re-derive from the shipped files.

What it does NOT prove: that the verdict is right, fair, or consistent. The label is one California DMV outcome, not ground truth about language. Agreement with a biased reviewer is still agreement with a biased reviewer. It also says nothing about plates unlike those in the corpus, or about any population other than the one the DMV happened to process.

Corrections that stand

This project's value is that it was measured honestly, so the errors stay in the record. A card that quietly dropped them would misrepresent the work.

1. The DECODE:-prefix scoring bug — an entire sweep read ~16 points too high. The evaluator called the grader's f1() on raw model output instead of stripping the DECODE: prefix the way the grader's own main() does. The literal token DECODE therefore matched on all 1,159 rows. The 7B read 64.36 when it is 48.24. Every arm in that sweep was inflated. What caught it was not the score — 64.36 looked plausible — but the adversarial floors: the spray floor came out 23.77 and the shuffled-gold floor 37.06 against recorded values of 5.61 and 1.27, and a shuffled-gold floor of 37 is impossible. A score can look plausible; a floor cannot lie. Both floors are recomputed and printed on every run in this repo, including for every quant.

2. The "100% mutation kill" was a small-sample artifact. The real figure is 99.8%. The originally published 1200/1200 kill rate was measured on a sample of mutants. Over the full 1,159-row test set (4,636 mutants) the strict grader kills 4,626/4,636 = 99.78%, with 0 false negatives. Ten mutants survive: one spray-attack and nine wrong-row mutants. Both records are in the log; the 99.8% is the one that counts.

3. The v1 self-selection artifact — an early decode score is RETRACTED. An early decode figure was computed only on the subset of rows the model had effectively selected for itself by choosing to answer, which flatters a model twice: it drops the rows it found hardest and keeps the comparison on its own favourable ground. Forced to answer on every row, almost all of the apparent advantage disappeared, and a "difficulty gap" derived from the same subset is the same artifact. Those figures are retracted and are deliberately not reproduced here — the v1-era per-row artifacts no longer exist on disk, so nothing in this repo could re-derive them, and republishing a retracted number from memory is exactly the habit this section exists to break. The retraction stands; the numbers stay in the project log.

4. "It's an oracle ceiling" was an inference, and measurement contradicted it. Four independent levers (more data, normalised targets, DPO, 2× scale) all landed inside noise of 48.24, and the first explanation offered was that the oracle had run out of room. Measured, that is mostly wrong: 29.0% of rows score exactly zero, but only 14.6% of those (4.2% of all rows) share a 4-character stem with the gold answer — 85.4% of zeros are genuine misses. It is a capability ceiling of this model family on this task, not a broken ruler. Loosening the oracle to stem-4 matching reads 51.66 (+3.42) but raises the spray floor 5.61 → 7.83, about 40% relative — so it was rejected. Never loosen an oracle to improve a number.

5. Inter-reviewer agreement here is UNMEASURABLE — do not quote a human ceiling. Only 2 of 23,307 distinct plates carry more than one DMV comment. The n=2 figure (19.44) is not a human baseline and must not be cited as one.

Arms that are NOT published, and why

plateclerk-b-14b (decode, 14B) — scored 48.94, +0.70 over the 7B with CI [-0.95, +2.34]: a null. It is not published because it costs 2× the parameters for a difference indistinguishable from noise. It is not evidence that scale fails, and it must not be cited as a capacity result in either direction.
verdict-3b-v2 (verdict, 3B) — degenerate. It emits DENIED on all 4,664 held-out rows at chance AUC 0.5030, and decisively it is degenerate on rows it trained on too (0/800 APPROVED, AUC 0.5352). A capacity ceiling cannot make a model fail its own training data, so this is a training failure, cause not isolated — recorded as UNRESOLVED, not as a scaling finding. Independently, Qwen/Qwen2.5-3B-Instruct is under the Qwen RESEARCH LICENSE, so a 3B derivative could not have been published here regardless. Also worth distrusting: its tuned-vs-base McNemar reads χ²=51.91 "SIGNIFICANT" — for a model that answers DENIED to everything. Its base was simply worse than a constant. Significance against a bad reference is not capability.

Training

Base: Qwen/Qwen2.5-1.5B-Instruct
Method: LoRA SFT, r=16, α=32, all attention + MLP projections
18799 training examples (sha256:f5c1690da1de4492...), 2 epochs, 8460 steps, peak LR 0.0002, final loss 0.0628
0 rows truncated (headCut=0, labelCut=0)
Held-out test rows were never trained on.

Licence and attribution

Checked against the actual licence files, not assumed — Qwen2.5 licences differ by size.

Base model: Qwen/Qwen2.5-1.5B-Instruct — Apache-2.0. Verified by fetching the base repo's own LICENSE; the 0.5B, 1.5B and 7B Instruct LICENSE files are byte-identical (sha256:832dd9e00a68dd83...) and are the Apache-2.0 text. Apache-2.0 permits redistribution of a derivative with attribution; the base LICENSE ships here as LICENSE.
Not all Qwen2.5 sizes are Apache-2.0. Qwen/Qwen2.5-3B-Instruct is released under the Qwen RESEARCH LICENSE, which is not an open redistribution licence. No 3B derivative is published here — see "Arms that are not published".
Training data: DarwinAnim8or/DMV-Plate-Review — MIT, confirmed from the dataset repo's metadata. Derived from California DMV personalised-plate application review records.
This repo: adapter and GGUFs released under Apache-2.0, matching the base.

What is published here

adapter/ — the LoRA adapter (r=16, α=32, all attention + MLP projections). This is the only thing that was actually trained.
gguf/ — the quant array: f16 (the unquantised control) plus the quants measured above.
eval/ — the grader(s), the exact held-out split, and the degradation table, so every number in this card can be re-derived rather than taken on trust.

Merged fp16 Hugging Face weights are NOT included. They are reconstructible from base + adapter, they triple the repo size for no new information, and the local merge step writes a degraded tokenizer_config.json that transformers cannot load back — publishing it would ship something nobody can use. Merge locally with peft if you want them.

Running it

The GGUFs are built for llama.cpp. Drive them through /completion with the prompt built from the chat template in adapter/chat_template.jinja (byte-identical to stock Qwen2.5's). Greedy decoding — every number in this card is temperature=0, top_k=1.

bash
llama-server -m gguf/verdict-15b-v2-Q5_K_M.gguf -c 4096

All measurements here were taken through /completion with a locally-built prompt string, not /v1/chat/completions. That is deliberate: on this hardware a server silently applying its own template has moved the same weights from 0.30% to 36.96% on an unrelated task. If you use a chat endpoint, verify your prompt matches the template before trusting any output.

Citation

bibtex
@misc{plateclerk2026,
  title  = {Plate Clerk — Verdict Head (1.5B)},
  author = {Blazing Customs},
  year   = {2026},
  url    = {https://huggingface.co/BlazingCustoms/plateclerk-verdict-1.5b},
  note   = {LoRA fine-tune of Qwen/Qwen2.5-1.5B-Instruct; evaluated by program oracle, no LLM judge}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

BlazingCustoms

Model Tree

Base

Qwen/Qwen2.5-1.5B-Instruct

Fine-tuned

this model

Input Modalities

Text

Output Modalities