AltronisSG

judgment-qc-gate-qwen3-4b-lora

README

License: apache-2.0

Design rule: skill, not facts

The corpus deliberately teaches how to judge and keeps facts out of the weights. Facts belong in retrieval, where they can be updated without retraining. A judgment model that memorises project details is stale in a month; one that learns "verify before claiming done" is not.

Output contract

Exactly one line, nothing else:

markdown
VERDICT: OK
VERDICT: <TAG> | <one-sentence correction>
VERDICT: ABSTAIN | <what is missing>

<TAG> is drawn from a closed vocabulary of 14: VERIFY, NO-OVERCLAIM, HOLD-LINE, SCOPE, SIMPLE, DESIGN, SAFETY, ROOT-CAUSE, NO-SILENT-FAIL, MAINTAIN, CURRENT, PROACTIVE, CONCERN-MAP, FULL-BUILD.

The closed vocabulary is the single most important design decision in this model. An earlier version used free-form tags: 85 distinct tags across 117 examples taught the model to invent them, and it produced recursive word-salad in production ("a claim about a claim is a claim about a claim"). Fixing the schema fixed the behaviour.

Training data

214 examples. Each is a real situation from months of AI-assisted engineering work, paired with the correction the operator actually gave at the time. Every example was individually reviewed and approved by that operator before entering the corpus — nothing is synthetic and nothing is bulk imported. 156 train, 58 held out for evaluation.

Examples are generalised: no client names, no project internals, no file paths, no figures, no identifiers. See Limitations for the one exception.

Evaluation — measured, not estimated

Numbers below were measured on 2026-07-25 against this exact checkpoint while it was serving live.

Table with columns: Check, Result
Check	Result
Known violations caught (`viol-probe`, n=12)	12 / 12
False positives on clean situations (`fp-probe`, n=30)	3 / 30 (10%)
Real agent turns judged in production (Stop hook, n=16)	16 / 16 passed, 0 blocks
Live spot check, 2 violations + 2 clean	4 / 4 correct
Held-out set, never trained on (n=58)	58/58 output contract, 0/16 false blocks, 33/34 caught, 12/34 exact tag

The three false positives are all over-eager VERIFY/DESIGN verdicts on replies that had already done the verification. One of them contradicts the operator's own delivery rule (it asked for risks before the outcome; the standard is outcome first, risks last). Those three are the training targets for the next version.

Recall is at ceiling on the current probe set, which means the probe set is too easy to measure against — a harder violation set is needed before recall can be improved honestly.

Base-model comparison (2026-07-25)

A Gemma-4-E4B sibling trained on the identical 156/58 split scores 53/58 on the output contract, 0/16 false blocks, 33/34 caught, 13/34 exact tag. Judgment is a wash — one tag apart on n=34 is noise. Qwen stays served because it never breaks the output contract: the Gemma variant emits its reasoning instead of a verdict on 5 of 58, mostly ABSTAIN cases, which any parser reads as a block.

Version history, kept honest

Table with columns: Version, Corpus, Outcome
Version	Corpus	Outcome
v2	117, free-form tags	Majority degenerate verdicts in live use; 1 genuinely useful catch across ~25 judgments
v3	117, closed schema	Degenerate output eliminated on a 6-case replay; over-eager on ambiguous input
v6	214, closed schema	3/30 false blocks, 12/12 caught. Blocked 0 of 16 real turns
v7a	178 train, 17% clean	Rejected. 6/30 false blocks — twice v6's rate, same recall
v7b	266 train, 11% clean (violations oversampled 5×)	13/30 false blocks (43%). Oversampling added zero clean examples and made the imbalance worse

For context on how far that came: in the ~9 hours before v6 went live, the same Stop hook blocked 75 of 85 agent turns, and MAINTAIN was being emitted as a catch-all 23 times.

The v7 → v8 result is the most useful thing in this table. v7a and v7b were trained on more violations and got measurably worse at recognising correct work — v7b blocked 43% of clean situations while still catching 12/12. Inspecting the failures showed a judge that had only ever been shown crimes: it blocked an agent for refusing to guess a CVE number, told an agent to paraphrase an error instead of quoting it verbatim, and twice emitted a verdict that agreed with the agent while still returning a block.

v8 is v7a's violation data plus 28 approved clean examples — situations where the work was done right. Nothing else changed. False positives halved. If you are distilling a judgment function, the lesson generalises: a discriminator needs the negative space. Train it only on failures and it learns that everything is a failure.

Usage

Serve the GGUF with llama-server and post a situation prefixed with its phase:

markdown
[phase: post] Assistant declared the fix deployed and working, but never ran the code
or looked at any output.

markdown
VERDICT: VERIFY | You said it was working, so you must have verified it.

phase is one of pre (plan / approach), during (execution), post (before it ships / claiming done). Exit-code use: treat VERDICT: OK as pass, anything else as a correction to act on, and fail open, loudly — if the gate is down, let the work through and say so rather than blocking it silently.

Note: the fine-tune emits an empty <think></think> block before its verdict (the corpus contains no reasoning traces). Strip it in post-processing, or serve with thinking disabled (--reasoning-budget 0).

Limitations

Narrow by design. It judges work-process situations phrased like its corpus. It is not a general reviewer, a linter, or a safety model.
Prose-only. It reads what an agent says. It cannot see what an agent did — the most important miss in its first day of production was a real mistake that happened in tool-space, invisible to a gate that only reads text.
The probe set flatters it. 12/12 recall means the violations were too obvious. Do not read that row as "catches everything".
Memorisation. A LoRA over 206 training examples will reproduce training phrasings when prompted in-domain. The corpus was curated to hold no client names, project internals, paths, figures, or identifiers, so this is a stylistic property rather than a privacy leak.
One disclosed exception to the above. 3 of the 206 training situations still contain the operator's given name. That name is the repository's own byline, so nothing private is revealed. (The system prompt names him in every row by design — it is the instruction that defines whose standards are being applied, not corpus content.) A scan of every field entering the training text found no client names, project names, paths, URLs, figures, phone numbers, addresses, or identifiers.
Single operator. These are one person's standards. That is the point of the model, and also the reason it should not be treated as a general measure of work quality.

Attribution

Built by Altronis. The corpus is one operator's own review history, approved example by example.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

AltronisSG

Model Tree

Base

Qwen/Qwen3-4B

Adapter

this model

Input Modalities

Text

Output Modalities