ThakiCloud

ChainCheck-Judge-Qwen3.8-27B

README

License: apache-2.0

Measurements (2026-10-05, conditions stated)

Same prompt, same scoring pipeline, one B200 GPU. Pairs a 27B model answers closed-book are excluded. Intervals are 95% pair-bootstrap (2,000 resamples). Σ = chain effect − |edit effect|; a lower bound above zero means the score reacts more to breaking the chain than to editing the passages.

Table with columns: Axis, zero-shot base, This model
Axis	zero-shot base	This model
2Wiki nominal AUC, real entities (n=295 pairs)	0.905	0.997
Chain selectivity Σ, MuSiQue test, real (n=324)	+0.201 [+0.123, +0.278]	+0.225 [+0.182, +0.269]
Chain selectivity Σ, MuSiQue confirmation, real (n=122)	+0.180 [+0.066, +0.295]	+0.107 [+0.057, +0.164]
Chain selectivity Σ, 2Wiki replication, real (n=295)	+0.129 [+0.054, +0.200]	+0.461 [+0.397, +0.485]
Chain selectivity Σ, MuSiQue test, synthetic (n=224)	+0.388 [+0.312, +0.420]	+0.125 [+0.085, +0.165]
Chain selectivity Σ, MuSiQue confirmation, synthetic (n=98)	+0.418 [+0.316, +0.469]	+0.051 [+0.010, +0.102]
Chain selectivity Σ, 2Wiki replication, synthetic (n=141)	+0.397 [+0.305, +0.454]	+0.170 [+0.106, +0.234]

Honest caveats:

The tuned model tends to score the consistently edited context (B) above the untouched one (A): the edit effect is negative in 5 of 6 cells (range -0.449 to +0.019). Σ subtracts |EE|, so this is already penalised in the table, but the score is not edit-invariant.
On real entities the tuned model is not better everywhere: chain selectivity is lower than zero-shot on MuSiQue confirmation (+0.180 → +0.107). The intervals overlap, so this is not a measured regression, but it is not a gain either.
With synthetic replacement entities, chain selectivity is lower after tuning than for the same-size zero-shot model on MuSiQue test (+0.388 → +0.125), MuSiQue confirmation (+0.418 → +0.051), 2Wiki replication (+0.397 → +0.170). The gains are on real entities; do not assume they transfer to entities the model has never seen.
2Wiki was never used for training or model selection, and MuSiQue confirmation is a held-out slice built after the protocol was frozen. The release gate (2Wiki Σ lower bound > 0 on real and synthetic, MuSiQue test real Σ lower bound > 0, 2Wiki nominal no more than 0.02 below zero-shot) was fixed before any score was seen.
The zero-shot column is the paper's reference judge re-scored through this pipeline. It ranks items essentially identically to the paper's run, but A and B differ by one entity and score almost the same, so small bf16 numerical differences flip some comparisons and Σ moves within the paper's reported interval.

Usage

python

import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer repo = "ThakiCloud/ChainCheck-Judge-Qwen3.8-27B"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto") SYSTEM = ("You are a strict evidence auditor for a retrieval-augmented QA system. "          "Decide whether the given passages, taken together, contain enough information "          "to fully answer the question. Related-but-insufficient passages do NOT count.")USER = ("Question:\n{query}\n\nPassages:\n{passages}\n\n"        "Do the passages together contain sufficient evidence to answer the question? "        "Answer with a single word: yes or no.") yes = [i for i in range(len(tok)) if tok.decode([i]).strip().lower() in {"yes", "y"}]no = [i for i in range(len(tok)) if tok.decode([i]).strip().lower() in {"no", "n"}] def sufficiency_score(query, passages):    body = "\n\n".join(f"[{i + 1}] {p[:4000]}" for i, p in enumerate(passages))    msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": USER.format(query=query, passages=body)}]    text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True, enable_thinking=False)    ids = tok(text, return_tensors="pt", add_special_tokens=False).to(model.device)    with torch.no_grad():        lp = torch.log_softmax(model(**ids).logits[0, -1].float(), -1)    return (torch.logsumexp(lp[yes], -1) - torch.logsumexp(lp[no], -1)).item()  # > 0 leans "sufficient"

The tokenizer, vocabulary and architecture are identical to the base (standard load). The score is a log-odds, not a calibrated probability: choose the threshold on your own validation data and use it to gate generation (answer, retrieve more, or abstain).

Limitations

English Wikipedia multi-hop questions only (MuSiQue, 2WikiMultiHopQA). The edits are entity substitutions on two-hop chains; numerical errors, temporal staleness and longer chains are not measured.
A positive Σ says the score reacts more to chain breaks than to edits on this benchmark. It does not certify that answers built on accepted contexts are correct.
Not a chat model. It is meant to be read through the yes/no logits shown above, not through free generation.

Benchmark

All three sizes, each next to its own zero-shot baseline:

Table with columns: Model, Set, Nominal (real), Σ real [95% CI], Σ synthetic [95% CI]
Model	Set	Nominal (real)	Σ real [95% CI]	Σ synthetic [95% CI]
Qwen3.5-4B zero-shot	MuSiQue test	0.889	+0.028 [-0.052, +0.111]	+0.188 [+0.094, +0.277]
Qwen3.5-4B zero-shot	MuSiQue confirmation	0.885	+0.090 [-0.049, +0.222]	+0.327 [+0.194, +0.378]
Qwen3.5-4B zero-shot	2Wiki replication	0.932	+0.136 [+0.058, +0.214]

Qwen3.5-4B: released
Qwen3.5-9B: released
Qwen3.8-27B: released

Benchmark kit and standalone evaluator: ThakiCloud/ChainCheck.

Training

LoRA (rank 8, α 16, attention q/k/v/o on full-attention layers, lr 1e-4, gradient accumulation 8, seed 0, one epoch over 7,481 MuSiQue train triplets) with labels A = 1, B = 1, D = 0 and loss = binary cross-entropy + max(0, 1 − (min(s_A, s_B) − s_D)). After merging, the full weights were reloaded from disk and re-scored on 64 held-out items to confirm they reproduce the adapter scores.

License

Apache-2.0 (base: Qwen/Qwen3.8-27B). Training and evaluation passages derive from MuSiQue (CC BY 4.0) and 2WikiMultiHopQA (Apache-2.0); passage text originates from Wikipedia.

Paper

ChainCheck: When Near-Perfect Evidence-Sufficiency Scores Fail to Distinguish Chain-Sensitive Systems — preprint forthcoming

The paper shows that near-perfect nominal evidence scores do not reliably distinguish chain-sensitive systems, and that separating chain sensitivity from edit sensitivity changes the apparent ordering of systems. This model applies that protocol as a training signal. Read its gains against the same-size zero-shot column, not against the nominal score alone, which is close to its ceiling for every system.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

ThakiCloud

Model Tree

Base

Qwen/Qwen3.8-27B

Fine-tuned

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality