BlazingCustoms

plateclerk-decode-7b

README

License: apache-2.0

Scope — binding

Entertainment and exploration only.
This is NOT a content-moderation system. It was never built, tuned or measured as one.
It must not be used to make decisions about real people — not applications, not eligibility, not enforcement, not screening.
It reproduces California DMV reviewers' readings, including their biases and their inconsistencies. It was trained to imitate a particular set of human decisions, not to be correct, fair or consistent. Where those reviewers disagreed with each other or with themselves, the model learned the disagreement.
It emits offensive strings by design, because reading offensive intent in obfuscated text is the task.

Anything outside that is out of scope and unmeasured.

Headline metric

token-F1 = 48.24 (v1 strict) / 48.81 (N1) on 1,159 held-out rows, bf16 reference (transformers, greedy), 0% abstention.

Every arm below was rescored from its stored per-row predictions with the corrected grader for this card. Each arm is compared to its own run's control, which is named — the arms do not all share one baseline, and pretending they do would repeat the very confound this card warns about.

Table with columns: arm, token-F1 (v1 strict), its own control, delta
arm	token-F1 (v1 strict)	its own control	delta
this model (7B SFT)	48.24	—	control
14B	48.94	48.24	+0.70, NULL — CI [-0.92, +2.37]
normalised targets	48.37	48.24	+0.13, null
DPO, length-balanced	48.07	48.39	-0.32, null
DPO, raw pairs	38.44	48.39	-9.95, regression
+57% more data (v3)

Two controls appear because two measurement lineages do. The DPO arms carry their own run's SFT reference (48.39). The v3 data arm belongs to an older lineage entirely — it and its 45.56 control were measured on a Q4_K_M GGUF through LM Studio, not bf16 through transformers, so 44.83 is not comparable to 48.24 and only its delta against 45.56 means anything. That is the same precision-and-serving-path confound this card warns about in the quantisation section; it is called out here rather than hidden behind a shared header.

Four independent levers — more data, better targets, RL, and 2× parameters — all landed inside noise of 48.24. That is a capability ceiling for this model family on this task, not four unlucky attempts, and not a broken oracle (see correction 4).

Where the remaining error is

69.9% of test rows are pure knowledge/slang and score 43.73 token-F1, forfeiting 39.32 of the 54.44 points lost. 74.4% require knowledge the plate string does not contain. The 0.5B→7B gain landed in exactly the scale-sensitive buckets (phonetic +11.65, foreign-language +20.15), which is why scale looked promising — and why the 14B's +0.70 null was the informative result.

Quantisation — measured, not assumed

Every quant was evaluated on the same held-out rows with the same grader as the reference. Quantisation is not assumed lossless.

The comparison is decomposed on purpose. Comparing a GGUF quant directly against the bf16 reference confounds two changes — precision and serving path (transformers vs llama.cpp). So the array carries an f16 GGUF control: bf16→f16 isolates the serving path, f16→QN isolates quantisation. Only the second is reported as a quantisation cost. The f16 GGUF reads 47.57 against the bf16 reference 48.24, so the serving path costs -0.67 and every delta below is genuinely quantisation.

The f16 GGUF is deliberately NOT shipped in this repo. The merged model.safetensors at the root already carries full precision, so an f16 GGUF would be the same numbers in a second container for another 15.2 GB, and it is reproducible from the merged weights with a single convert_hf_to_gguf.py call. The f16 measurements below stay — the control is a measurement, not an artifact anyone needs to download. The 0.5B and 1.5B repos do ship theirs, where it is cheap.

Table with columns: variant, size, shipped?, token-F1 (v1 strict), Δ vs f16, 95% CI (paired), N1, spray floor, shuffled floor, empty
variant	size	shipped?	token-F1 (v1 strict)	Δ vs f16	95% CI (paired)	N1	spray floor	shuffled floor	empty
f16	15.24 GB	no — control only	47.57	—	reference	48.05	5.61	1.27	0

CIs are paired bootstraps over the same rows (10,000 resamples for token-F1, 2,000 for AUC). The floors are recomputed for every variant, never inherited.

This measurement overturns a number this project previously published about itself. The prior record read a Q4_K_M GGUF at 45.56 against a bf16 reference of 48.24 and called the ~2.7-point gap the cost of quantisation. Decomposed here, it is not: the same quant type on the same 1,159 rows with the same grader reads 48.04, and quantisation is worth +0.47 [-0.55, +1.47] against the f16 control — indistinguishable from zero. Essentially all of the original gap was the serving path, not the weights: the old figure came from a chat endpoint, while these are /completion calls carrying the exact prompt the model was trained on.

Note also that the quants here are non-monotonic — Q4_K_M scores above f16. That is the signature of noise, not of a quant improving a model, and it is reported as such. Every quant's CI straddles zero; none of these differences is real.

Artifact digests

f16 — sha256:310c57f02d85c57d790b4c89652a6c998c3872c5d1e6d3a440ea715d1f990b67 (15.24 GB)
Q8_0 — sha256:78ff5e82c878b1977ecbd0791821d11722602eb534ad2063a6a5edd36d2c9ca6 (8.10 GB)
Q5_K_M — sha256:e3bdf591ac9a02fc1f215ee51a7d1829911cc4f8ec10f6b7191ed751c97b051e (5.44 GB)
Q4_K_M — sha256:39cf5a02ce7227d0bba45b0e60827baf394991f796cdc2d1374c7e60bc85a294 (4.68 GB)

The oracle, and what it does NOT prove

Grading is done by a program, never by an LLM judge.

Headline grader: eval/plateclerk_decode_v1_strict.py — sha256:24b3728d0baac19a. Token-level F1 between the model's decode and the DMV reviewer's recorded reading, after stopword filtering.
Second grader reported always: eval/plateclerk_decode.py (N1) — sha256:e0b9feae58f43c27, which fixes an asymmetric stopword-fallback bug. N1 is adversarially identical to v1 strict: same kills, same 10 survivors, 0/2,088 false negatives. Both numbers are reported forever; v1 strict is the headline.
Mutation-tested against scripts/mutations/plateclerk-decode.py (sha256:f0265b926fc5015b): 4,636 mutants, 4,626 killed = 99.78%, 0 false negatives, 10 survivors (one spray-attack, nine wrong-row).
Held-out split: eval/dmv-plateclerk-decode-v2_test.jsonl, 1,159 rows, sha256:c1a645c8f5fce4b3....
Adversarial floors, recomputed on every run: spray 5.61 (one fixed offensive string for every row) and (real answers, wrong rows). A score that does not stand well clear of both means nothing. Note that here — adjacent test rows share decodes and it reads 10.06 instead of ~1.2.

What it proves: token overlap with what a DMV reviewer actually wrote, on unseen plates, re-derivable from the shipped files.

What it does NOT prove: that the decode is the only correct reading, or that a low-scoring output is wrong — token-F1 punishes correct paraphrase. 25.3% of gold tokens are words the grader deletes as stopwords, and 44.1% of rows contain at least one. It is a consistent ruler, not a semantic one.

Corrections that stand

This project's value is that it was measured honestly, so the errors stay in the record. A card that quietly dropped them would misrepresent the work.

1. The DECODE:-prefix scoring bug — an entire sweep read ~16 points too high. The evaluator called the grader's f1() on raw model output instead of stripping the DECODE: prefix the way the grader's own main() does. The literal token DECODE therefore matched on all 1,159 rows. The 7B read 64.36 when it is 48.24. Every arm in that sweep was inflated. What caught it was not the score — 64.36 looked plausible — but the adversarial floors: the spray floor came out 23.77 and the shuffled-gold floor 37.06 against recorded values of 5.61 and 1.27, and a shuffled-gold floor of 37 is impossible. A score can look plausible; a floor cannot lie. Both floors are recomputed and printed on every run in this repo, including for every quant.

2. The "100% mutation kill" was a small-sample artifact. The real figure is 99.8%. The originally published 1200/1200 kill rate was measured on a sample of mutants. Over the full 1,159-row test set (4,636 mutants) the strict grader kills 4,626/4,636 = 99.78%, with 0 false negatives. Ten mutants survive: one spray-attack and nine wrong-row mutants. Both records are in the log; the 99.8% is the one that counts.

3. The v1 self-selection artifact — an early decode score is RETRACTED. An early decode figure was computed only on the subset of rows the model had effectively selected for itself by choosing to answer, which flatters a model twice: it drops the rows it found hardest and keeps the comparison on its own favourable ground. Forced to answer on every row, almost all of the apparent advantage disappeared, and a "difficulty gap" derived from the same subset is the same artifact. Those figures are retracted and are deliberately not reproduced here — the v1-era per-row artifacts no longer exist on disk, so nothing in this repo could re-derive them, and republishing a retracted number from memory is exactly the habit this section exists to break. The retraction stands; the numbers stay in the project log.

4. "It's an oracle ceiling" was an inference, and measurement contradicted it. Four independent levers (more data, normalised targets, DPO, 2× scale) all landed inside noise of 48.24, and the first explanation offered was that the oracle had run out of room. Measured, that is mostly wrong: 29.0% of rows score exactly zero, but only 14.6% of those (4.2% of all rows) share a 4-character stem with the gold answer — 85.4% of zeros are genuine misses. It is a capability ceiling of this model family on this task, not a broken ruler. Loosening the oracle to stem-4 matching reads 51.66 (+3.42) but raises the spray floor 5.61 → 7.83, about 40% relative — so it was rejected. Never loosen an oracle to improve a number.

5. Inter-reviewer agreement here is UNMEASURABLE — do not quote a human ceiling. Only 2 of 23,307 distinct plates carry more than one DMV comment. The n=2 figure (19.44) is not a human baseline and must not be cited as one.

Arms that are NOT published, and why

plateclerk-b-14b (decode, 14B) — scored 48.94, +0.70 over the 7B with CI [-0.95, +2.34]: a null. It is not published because it costs 2× the parameters for a difference indistinguishable from noise. It is not evidence that scale fails, and it must not be cited as a capacity result in either direction.
verdict-3b-v2 (verdict, 3B) — degenerate. It emits DENIED on all 4,664 held-out rows at chance AUC 0.5030, and decisively it is degenerate on rows it trained on too (0/800 APPROVED, AUC 0.5352). A capacity ceiling cannot make a model fail its own training data, so this is a training failure, cause not isolated — recorded as UNRESOLVED, not as a scaling finding. Independently, Qwen/Qwen2.5-3B-Instruct is under the Qwen RESEARCH LICENSE, so a 3B derivative could not have been published here regardless. Also worth distrusting: its tuned-vs-base McNemar reads χ²=51.91 "SIGNIFICANT" — for a model that answers DENIED to everything. Its base was simply worse than a constant. Significance against a bad reference is not capability.

Training

Base: Qwen/Qwen2.5-7B-Instruct
Method: LoRA SFT, r=16, α=32, all attention + MLP projections
10784 training examples (sha256:cfefc11d6e19abc3...), 2 epochs, 4854 steps, peak LR 0.0002, final loss 0.4735
0 rows truncated (headCut=0, labelCut=0)
Held-out test rows were never trained on.

Licence and attribution

Checked against the actual licence files, not assumed — Qwen2.5 licences differ by size.

Base model: Qwen/Qwen2.5-7B-Instruct — Apache-2.0. Verified by fetching the base repo's own LICENSE; the 0.5B, 1.5B and 7B Instruct LICENSE files are byte-identical (sha256:832dd9e00a68dd83...) and are the Apache-2.0 text. Apache-2.0 permits redistribution of a derivative with attribution; the base LICENSE ships here as LICENSE.
Not all Qwen2.5 sizes are Apache-2.0. Qwen/Qwen2.5-3B-Instruct is released under the Qwen RESEARCH LICENSE, which is not an open redistribution licence. No 3B derivative is published here — see "Arms that are not published".
Training data: DarwinAnim8or/DMV-Plate-Review — MIT, confirmed from the dataset repo's metadata. Derived from California DMV personalised-plate application review records.
This repo: adapter and GGUFs released under Apache-2.0, matching the base.

What is published here

adapter/ — the LoRA adapter (r=16, α=32, all attention + MLP projections). This is the only thing that was actually trained.
gguf/ — the quant array: f16 (the unquantised control) plus the quants measured above.
eval/ — the grader(s), the exact held-out split, and the degradation table, so every number in this card can be re-derived rather than taken on trust.

Merged fp16 Hugging Face weights are NOT included. They are reconstructible from base + adapter, they triple the repo size for no new information, and the local merge step writes a degraded tokenizer_config.json that transformers cannot load back — publishing it would ship something nobody can use. Merge locally with peft if you want them.

Running it

The GGUFs are built for llama.cpp. Drive them through /completion with the prompt built from the chat template in adapter/chat_template.jinja (byte-identical to stock Qwen2.5's). Greedy decoding — every number in this card is temperature=0, top_k=1.

bash
llama-server -m gguf/plateclerk-b-7b-Q5_K_M.gguf -c 4096

All measurements here were taken through /completion with a locally-built prompt string, not /v1/chat/completions. That is deliberate: on this hardware a server silently applying its own template has moved the same weights from 0.30% to 36.96% on an unrelated task. If you use a chat endpoint, verify your prompt matches the template before trusting any output.

Citation

bibtex
@misc{plateclerk2026,
  title  = {Plate Clerk — Decode Head (7B)},
  author = {Blazing Customs},
  year   = {2026},
  url    = {https://huggingface.co/BlazingCustoms/plateclerk-decode-7b},
  note   = {LoRA fine-tune of Qwen/Qwen2.5-7B-Instruct; evaluated by program oracle, no LLM judge}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

BlazingCustoms

Model Tree

Base

Qwen/Qwen2.5-7B-Instruct

Fine-tuned

this model

Input Modalities

Text

Output Modalities