GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
narcolepticchicken
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
narcolepticchicken
Model Tree
Qwen/Qwen2.5-7B-Instruct
Input Modalities
Output Modalities
Supported Functionality
| Model | Test split | Agreement | False-pass | False-fail |
|---|---|---|---|---|
| v1 (this model) | v1-test (score≥4→PASS) | 83.5% | 29.8% | 3.3% |
| v2 | v1-test | 80.4% | 23.5% | 15.7% |
| v2 | v2-test (hard negatives) | 69.3% | 39.5% | 6.7% |
v1's weakness: a 29.8% false-pass rate — it is too lenient on borderline gold-FAIL responses (it internally re-scores 2–3 up to 4–5 and passes them). The v2 de-biasing recipe (stricter PASS threshold + hard negatives) reduces this to 23.5% on the matched split. A threshold sweep on the v2 model proved the residual false-pass on adversarial fluent-but-wrong responses is not a calibration issue (min achievable ~39%), motivating a contrastive DPO stage (v3). See the v2 dataset repo for the full pipeline and eval scripts.
python
from transformers import AutoModelForCausalLM, AutoTokenizerimport torch model_id = "narcolepticchicken/qwen2.5-7b-verifier"tok = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="cuda") SYSTEM = ("You are a strict verifier. Decide whether the candidate response " "satisfies the given criterion. Write a short critique, then output " "exactly 'Verdict: PASS' or 'Verdict: FAIL'.")user = ("Task: ...\nCriterion: ...\nScore-5 description: ...\nScore-1 description: ...\n" "Reference answer: ...\nCandidate response: ...")msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": user}]ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to("cuda")out = model.generate(ids, max_new_tokens=512, do_sample=False)print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))Full SFT (not LoRA), assistant_only_loss=True, bf16, gradient checkpointing,
cosine schedule, lr 1e-5. TRL 1.5.1 · Transformers 5.9.0 · PyTorch 2.6.0.
GenRM 2408.15240 · Prometheus 2 2405.01535 · OffsetBias 2407.06551 · Master-RM 2507.08794 · RISE-Judge 2502.11689
Generated by ML Intern.
from transformers import AutoModelForCausalLM, AutoTokenizerimport torch model_id = "narcolepticchicken/qwen2.5-7b-verifier"tok = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="cuda") SYSTEM = ("You are a strict verifier. Decide whether the candidate response " "satisfies the given criterion. Write a short critique, then output " "exactly 'Verdict: PASS' or 'Verdict: FAIL'.")user = ("Task: ...\nCriterion: ...\nScore-5 description: ...\nScore-1 description: ...\n" "Reference answer: ...\nCandidate response: ...")msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": user}]ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to("cuda")out = model.generate(ids, max_new_tokens=512, do_sample=False)print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))