GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
narcolepticchicken
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
narcolepticchicken
Model Tree
Qwen/Qwen2.5-7B-Instruct
Input Modalities
Output Modalities
Supported Functionality
| Variant | PASS definition | Extra training signal |
|---|---|---|
| v1 | score ≥ 4 (of 5) | balanced PASS/FAIL |
| v2 (this model) | score == 5 (stricter) | + fluent-but-wrong hard negatives (OffsetBias 2407.06551, Master-RM 2507.08794) + length-balancing (RISE-Judge 2502.11689) |
The v2 recipe targets the v1 failure mode: v1 was too lenient on borderline gold-FAIL items (it re-scored 2–3 up to 4–5 and passed them).
| Model | Test split | Agreement | False-pass | False-fail |
|---|---|---|---|---|
| v1 | v1-test (score≥4→PASS) | 83.3% | 28.1% | 5.3% |
| v2 | v1-test (score≥4→PASS) | 80.4% | 23.5% | 15.7% |
| v2 | v2-test (score==5→PASS + hard neg) | 69.3% | 39.5% | 6.7% |
Reading the results. The de-biasing recipe did lower false-pass on the matched distribution (28.1% → 23.5% on v1-test) — the intended effect. But:
score==5→PASS target makes the model reject many acceptable
responses. This is a precision/recall trade, not a free lunch.Implication: SFT alone does not solve the fluent-but-wrong failure mode. The decision threshold and/or further data work (more / harder negatives, or a preference stage) are the next levers — see the threshold-sweep harness in the v2 dataset repo.
The model expects the verifier system prompt + a structured user message. The
test datasets carry ready-made messages; reuse that exact format.
python
from transformers import AutoModelForCausalLM, AutoTokenizerimport torch model_id = "narcolepticchicken/qwen2.5-7b-verifier-v2"tok = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="cuda") SYSTEM = ("You are a strict verifier. Decide whether the candidate response " "satisfies the given criterion. Write a short critique, then output " "exactly 'Verdict: PASS' or 'Verdict: FAIL'.")user = ( "Task: <the instruction the response was answering>\n" "Criterion: <the rubric to judge against>\n" "Score-5 description: <what a perfect response looks like>\n" "Score-1 description: <what a failing response looks like>\n" "Reference answer: <optional gold answer>\n" "Candidate response: <the response to verify>")msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": user}]ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to("cuda")out = model.generate(ids, max_new_tokens=512, do_sample=False)print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))For continuous scoring (recommended), read the probability of the PASS token
at the verdict position to get P(PASS), then threshold it to hit a target
false-pass rate. See eval_threshold_sweep.py in the dataset repo.
Full SFT (not LoRA), assistant_only_loss=True, bf16, gradient checkpointing,
cosine schedule, lr 1e-5, on narcolepticchicken/verifier-debias-v2.
This model repository was generated by ML Intern.
from transformers import AutoModelForCausalLM, AutoTokenizerimport torch model_id = "narcolepticchicken/qwen2.5-7b-verifier-v2"tok = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="cuda") SYSTEM = ("You are a strict verifier. Decide whether the candidate response " "satisfies the given criterion. Write a short critique, then output " "exactly 'Verdict: PASS' or 'Verdict: FAIL'.")user = ( "Task: <the instruction the response was answering>\n" "Criterion: <the rubric to judge against>\n" "Score-5 description: <what a perfect response looks like>\n" "Score-1 description: <what a failing response looks like>\n" "Reference answer: <optional gold answer>\n" "Candidate response: <the response to verify>")msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": user}]ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to("cuda")out = model.generate(ids, max_new_tokens=512, do_sample=False)print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))