GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
rotalabs
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
rotalabs
Model Tree
Qwen/Qwen2.5-7B-Instruct
Input Modalities
Output Modalities
Supported Functionality
| Metric | Result |
|---|---|
| Agreement | 0.953 (gate ≥ 0.85) |
| ECE | 0.013 (gate ≤ 0.10) |
| Per language | en 0.983 / hi 0.953 / hinglish 0.928 |
Per perturbation family: number_swap 1.00, negation 1.00, fabricated_fact 1.00, unsupported_claim 1.00 — and entity_swap 0.54, reported deliberately: the judge misses wrong-person/wrong-product substitutions about half the time. That is the v1 training priority. Score answers whose risk is entity identity accordingly.
python
import torchfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer base = AutoModelForCausalLM.from_pretrained( "Qwen/Qwen2.5-7B-Instruct", torch_dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(base, "rotalabs/sakshi-judge-indic")tok = AutoTokenizer.from_pretrained("rotalabs/sakshi-judge-indic") SYSTEM = ("You are a groundedness judge for banking documents. Given a CONTEXT " "and an ANSWER, decide whether every claim in the answer is supported " "by the context. Reply with exactly one word: grounded or hallucinated.")prompt = tok.apply_chat_template( [{"role": "system", "content": SYSTEM}, {"role": "user", "content": "CONTEXT:\n...\n\nANSWER:\n...\n\nVerdict:"}], add_generation_prompt=True, return_tensors="pt").to(model.device)logits = model(prompt).logits[0, -1]g = tok("grounded", add_special_tokens=False)["input_ids"][0]h = tok("hallucinated", add_special_tokens=False)["input_ids"][0]probs = torch.softmax(torch.stack([logits[g], logits[h]]).float(), dim=0)# probs[0] = P(grounded), probs[1] = P(hallucinated) — calibrated100% synthetic (context, answer, verdict) triples: banking passages rendered from structured facts; hallucinations produced by perturbing those facts (numbers, entities, negation) or injecting fabricated/unsupported claims — labels correct by construction, no distillation, no label noise. The gate set is construction-verified; human-verified evaluation is the v1 step. Trained on short factual banking passages — long multi-document contexts, tables, and reasoning-heavy answers are untested. A judge is a signal, not a verdict: route low-confidence or hallucinated outputs to human review.
Released by RotaVision under the rotalabs open-source commons, Apache-2.0. Generator, training and gated evaluation scripts live in the Sakshi repository.
import torchfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer base = AutoModelForCausalLM.from_pretrained( "Qwen/Qwen2.5-7B-Instruct", torch_dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(base, "rotalabs/sakshi-judge-indic")tok = AutoTokenizer.from_pretrained("rotalabs/sakshi-judge-indic") SYSTEM = ("You are a groundedness judge for banking documents. Given a CONTEXT " "and an ANSWER, decide whether every claim in the answer is supported " "by the context. Reply with exactly one word: grounded or hallucinated.")prompt = tok.apply_chat_template( [{"role": "system", "content": SYSTEM}, {"role": "user", "content": "CONTEXT:\n...\n\nANSWER:\n...\n\nVerdict:"}], add_generation_prompt=True, return_tensors="pt").to(model.device)logits = model(prompt).logits[0, -1]g = tok("grounded", add_special_tokens=False)["input_ids"][0]h = tok("hallucinated", add_special_tokens=False)["input_ids"][0]probs = torch.softmax(torch.stack([logits[g], logits[h]]).float(), dim=0)# probs[0] = P(grounded), probs[1] = P(hallucinated) — calibrated