Honest status
The prompt-ceiling gate for this spec fails. A two-family ablation
(gpt-5.5 and claude-sonnet-5, three strategies, 36 scenarios, cross-family
judged) puts the best prompted cell at 1.000, against a bar requiring it to
sit below 0.90. Prompting already solves this contract, so this adapter is a
diagnostic checkpoint, not evidence that data instilled a hard behaviour.
Two further caveats a reader needs:
- Train/eval share fixtures. The training seeds and the v1 eval set were
generated from the same defect templates with different filenames. Exact-hash
disjointness passes and proves nothing. A disjoint set (
review_v2) exists.
- Layer-1 only. The numbers below are the deterministic schema check.
Groundedness was not measured.
Table with columns: run, schema-pass, pressure+adversarial, false-clean, rule-recall| run | schema-pass | pressure+adversarial | false-clean | rule-recall |
|---|
| Qwen3-0.6B base | 0.028 | 0.042 | 0.045 | 0.000 |
| this adapter | 0.722 | 0.750 | 0.091 | 1.000 |
| gpt-5.5 prompted | 1.000 | 1.000 | 0.000 | 1.000 |
9 of the 10 remaining failures are LINE_NOT_IN_DIFF: the model learned the
JSON shape and the rule names, not how to count hunk lines.
Training
LoRA (not QLoRA — bf16, no quantisation), r=256, alpha=256,
target_modules="all-linear", lr 2e-4, 2 epochs, effective batch 16, on 414
teacher-distilled and filtered examples. Loss 2.35 -> 0.03.
Sibling checkpoints n207, n103, n51 form the data-efficiency curve.
Use
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B")
model = PeftModel.from_pretrained(base, "elguantletai002/checklist-reviewer-qwen3-0.6b-n414").merge_and_unload()
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")
The system prompt is spec/behavior_spec.md from the source repo. Behaviour is
undefined without it.