Behavior Spec
Given a question, retrieved context, and a candidate response, emit FAIL iff the
response makes at least one factual claim that is unsupported by or contradicts the
retrieved context — truth in the real world is irrelevant (strict grounding); PASS
otherwise, including responses that explicitly decline to answer for lack of context.
Judged at response level. Unsupported-but-true world knowledge is FAIL. Abstention is PASS.
Results
Full held-out set: 8,295 human-annotated items (RAGTruth test + FaithBench + HaluBench
with the HaluEval subset excised). No LLM judged these — the labels are human.
Table with columns: spec adherence, robustness (worst set), F1(FAIL), recall(FAIL), AUROC | spec adherence | robustness (worst set) | F1(FAIL) | recall(FAIL) | AUROC |
|---|
| base Qwen3-1.7B, zero-shot | 0.5561 | 0.4953 | 0.4979 | 0.4918 | 0.5669 |
| this model | 0.7180 | 0.5766 | 0.6938 | 0.7300 | 0.7846 |
Spec adherence = pooled balanced accuracy. Robustness = worst per-set balanced accuracy.
Against frontier models, same 60 hard scenarios
Scenarios are disagreement-mined: FaithBench ships 8 detector scores per item, ranked by
variance across them (selected items ~2x the corpus median disagreement).
Table with columns: model, spec adherence, recall(FAIL), specificity(PASS)| model | spec adherence | recall(FAIL) | specificity(PASS) |
|---|
| Claude Opus 5, chain-of-thought | 0.750 | 0.800 | 0.700 |
| GPT-5.6, few-shot | 0.717 | 0.800 | 0.633 |
| GPT-5.6, zero-shot | 0.700 | 0.800 | 0.600 |
| this model (1.7B) | 0.683 | 0.533 | 0.833 |
| Claude Opus 5, zero-shot | 0.633 | 0.867 | 0.400 |
| base Qwen3-1.7B | 0.533 | 0.100 | 0.967 |
n=60, so a 95% interval is roughly +/-0.12. The honest claim is parity-region, not victory.
The error profiles are opposites. Prompted frontier judges run high recall / low
specificity — they catch hallucinations and also reject grounded answers. This model
inverts it: specificity 0.833, recall 0.533. For a guardrail gating production traffic,
false alarms are the expensive error, which favours this profile — but see Limitations.
Usage
The prompt must be rendered exactly as in training or the numbers do not transfer.
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel BASE = "Qwen/Qwen3-1.7B"tok = AutoTokenizer.from_pretrained(BASE)model = PeftModel.from_pretrained( AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16, device_map="auto"), "aaryand/qwen3-1.7b-context-adherence-guardrail").eval() RUBRIC = """You are a groundedness guardrail for retrieval-augmented generation. \Judge whether the RESPONSE is fully supported by the CONTEXT. Rules:- Verdict FAIL if the response makes at least one factual claim that is unsupported \by or contradicts the context. Whether a claim is true in the real world is \irrelevant: a claim absent from the context is still unsupported (FAIL).- Verdict PASS otherwise. A response that declines to answer or says the context \lacks the information is faithful (PASS).""" def judge(question, context, response): body = (f"QUESTION: {question}\n\n" if question else "") + \ f"CONTEXT:\n{context[:24000]}\n\nRESPONSE:\n{response}" prompt = f"{RUBRIC}\n\n{body}\nAnswer with exactly one word: PASS or FAIL." ids = tok.apply_chat_template([{"role": "user", "content": prompt}], add_generation_prompt=True, return_tensors="pt", enable_thinking=False).to(model.device) with torch.no_grad(): logits = model(ids).logits[0, -1] p = torch.softmax(logits[[tok.encode("PASS")[0], tok.encode("FAIL")[0]]].float(), -1) return ("FAIL" if p[1] > p[0] else "PASS"), p[1].item() # verdict, P(FAIL) print(judge("Who wrote it?", "The report was written by Dr. Chen in 2019.", "Dr. Chen wrote it in 2019, and she also won a Nobel Prize."))# ('FAIL', 0.9...) — the Nobel claim is absent from the context
Training
QLoRA NF4, LoRA r=16 / alpha=32 on all-linear targets, 2 epochs, lr 1e-4 cosine,
effective batch 16, max 4096 tokens, one A100-40GB. Data: 10,710 items, 50/50 PASS/FAIL —
see the dataset.
The verdict token is not fed as input: the model sees the prompt and the loss is
cross-entropy at the final position against PASS/FAIL. That is bit-for-bit the quantity
the eval scores, so training and evaluation are the same measurement rather than two that
merely agree.
Minimum viable dataset size ~ 4,000. Marginal return per doubling: +0.017 (250->1k),
+0.048 (1k->4k), then +0.0088 (4k->10,710). This checkpoint uses all 10,710 because it is
the best measured model, but the curve stops paying for itself near 4,000.
Limitations
- Robustness gains less than half the headline. +0.081 worst-set against +0.162 pooled.
Better on average, only modestly better on the distribution it handles worst. Size
deployment risk from the robustness number.
- Under-flags. Recall 0.53 on the hard scenarios — it misses roughly half of subtle
hallucinations there. A precision instrument, not a safety net.
- One metric, one context. Grounding only. Says nothing about toxicity, prompt
injection, or tool selection.
- In-domain parity is not general-judge parity. Trained on RAGTruth-train; RAGTruth-test
shares its annotation conventions and generator distribution.
- Not calibrated. ECE was never computed. Treat P(FAIL) as an ordering, not a probability.
- No adversarial evaluation. Untested against jailbreaks or over-defense probes.
- Single seed. Seed variance was not measured; curve differences below ~0.01 are noise.
Reproduce
pip install torch transformers peft acceleratepython eval.py --model aaryand/qwen3-1.7b-context-adherence-guardrail --baselinepython eval.py --model aaryand/qwen3-1.7b-context-adherence-guardrail --eval-set your_set.jsonl
Code, raw transcripts for all 19 eval runs, ablation scripts and training logs are in the
project repository. Eval-code commit: b5467d033690f3909cfacfd69bfb3fa04cc6a946.