elguantletai002

checklist-reviewer-qwen3-0.6b-v3-n311

README

License: apache-2.0

Results

Judged by claude-sonnet-5, frozen rubric, 36 scenarios on review_v2 — fixtures share no distinctive line with the training seeds (test-enforced).

Table with columns: base, this adapter, prompted frontier ceiling
	base	this adapter	prompted frontier ceiling
spec-adherence	0.139	0.778	0.861
robustness (pressure + adversarial)	0.208	0.833	0.875
strict (right rule, legal anchor)	0.028	0.472	0.806
false-clean (lower better)	0.125	0.156	0.000

It does not reach the 0.90/0.90 reliability bar. Closer than any previous checkpoint, but the behaviour is improved rather than reliably instilled, and no minimum viable dataset size has been measured.

What changed from v2

Two defects fixed in one run, so the gain cannot be attributed to either alone:

Multi-turn examples added. Half the eval set carries a prior refusal and the v2 training set had none, so robustness was partly measuring an untrained behaviour. 168 records were added where the model has already answered and the user pushes back.
Loss no longer covers the system prompt. Converted to prompt/completion form so TRL masks the prompt; previously most of the gradient went into copying the ~1.1k-token spec back.

Against v2: adherence 0.611 → 0.778, robustness 0.583 → 0.833, strict 0.222 → 0.472.

Training

LoRA (not QLoRA — bf16, no quantisation; 4-bit saves VRAM that is not scarce at 0.6B). r=256, alpha=256, target_modules="all-linear", lr 2e-4, 2 epochs, effective batch 16, 311 examples from a nested seeded subset of 623.

Sibling checkpoints at N=77/155/623 form the data-efficiency curve. Note N=77 shows the table's highest robustness (0.875) and is not a good checkpoint: false-clean 0.281 means it buys the score with silence.

Use

python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B")
model = PeftModel.from_pretrained(base, "elguantletai002/checklist-reviewer-qwen3-0.6b-v3-n311").merge_and_unload()
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")

System prompt is spec/behavior_spec.md (v2) from the source repo. Behaviour is undefined without it.

Supersedes …-n455 (single-turn data, loss over the system prompt) and the n414/n207/n103/n51 family (trained on data that shared fixtures with its own eval set).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

elguantletai002

Model Tree

Base

Qwen/Qwen3-0.6B

Adapter

this model

Input Modalities

Text

Output Modalities