GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
sarpaydemir
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
sarpaydemir
Model Tree
Qwen/Qwen3.5-2B
Input Modalities
Output Modalities
Supported Functionality
Scored on 300 test hunks from five held-out repositories (56 high). The answer key is two frontier models' agreement, with a third model settling the 23 hunks they disagree on. The results page has every system and table.
| System | Recall | Precision | F1 | Cost / 1k hunks |
|---|---|---|---|---|
| This adapter, P(high) ≥ 0.148 | 0.91 | 0.55 | 0.69 | $0.036 |
| This adapter, its own answer | 0.64 | 0.77 | 0.70 | $0.036 |
| Kimi K3.1 Pro (ceiling) | 0.91 | 0.66 | 0.77 | $1.19 |
| DeepSeek V4.1 Flash (teacher) | 0.79 | 0.85 | 0.81 | $0.025 |
| Keyword/path heuristic | 0.73 | 0.35 | 0.48 | $0 |
high, on 300 hunks from five other repositories.high hunks), 100% valid JSON, 33× cheaper than the ceiling, and F1 7.8 points below the ceiling's, where the bar allows 10. Recall is one hunk above the line: had the 23 disputed hunks been settled by one of the two judges instead, it could be 0.89. On those hunks the adapter and the ceiling make the same number of mistakes, which narrows the F1 gap; on the other 277 it is 10.4 points.The input is the file path, a newline, then the hunk, under a short system prompt. Thinking is off. risk is the first field of the answer, so P(high) is one forward pass away.
python
import torchfrom peft import PeftModelfrom transformers import AutoModelForImageTextToText, AutoTokenizer repo = "sarpaydemir/hunkrisk-qwen3.5-2b-lora"THRESHOLD = 0.148 # 0.90 recall on validation; 0.5 follows the model's own answer # Qwen3.5 is multimodal. AutoModelForCausalLM builds a text-only variant with different module# names, and the adapter would silently fail to attach to it.base = AutoModelForImageTextToText.from_pretrained( "Qwen/Qwen3.5-2B", dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(base, repo).eval()tokenizer = AutoTokenizer.from_pretrained(repo) SYSTEM = ( "Classify the diff hunk for code review risk. Reply with one JSON object: " '{"risk": "high" or "low", "categories": [...], "reason": "<one sentence>"}. ' 'Categories: auth, money, schema, data, secrets; use ["none"] when risk is low.')file = "app/Http/Controllers/InvoiceController.php"hunk = """@@ -40,9 +40,6 @@ public function destroy(Request $request, Invoice $invoice)- if ($invoice->user_id !== $request->user()->id) {- abort(403);- } $invoice->delete();""" messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": f"{file}\n{hunk}"}]prompt = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, enable_thinking=False) # P(high): score the risk token right after the answer prefix.high, low = (tokenizer.encode(t, add_special_tokens=False)[0] for t in ("high", "low"))inputs = tokenizer(prompt + '{"risk": "', return_tensors="pt", add_special_tokens=False)with torch.inference_mode(): logits = model(**inputs.to(model.device)).logits[0, -1]p_high = torch.softmax(logits[[high, low]].float(), dim=0)[0].item()print(f"P(high) = {p_high:.3f}, flag = {p_high >= THRESHOLD}") # The full answer, with categories and reason.inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)output = model.generate(**inputs, max_new_tokens=160, do_sample=False)print(tokenizer.decode(output[0, inputs.input_ids.shape[1] :], skip_special_tokens=True))The evaluation ran with Unsloth's kernels, so scores from plain transformers can differ in the second decimal. From the repository, git diff | uv run python -m hunkrisk.predict --model sarpaydemir/hunkrisk-qwen3.5-2b-lora scores every hunk of a diff.
Trained on 5,735 hunks from 20 repositories, labeled by the teacher (18% high), and validated on 300 hunks from five other repositories. It ran 718 steps (two epochs) in 62 minutes. Validation loss was still falling at the end (0.588), so the published weights are the last checkpoint.
Single hunks from 30 permissively licensed open-source applications (Laravel, Django, FastAPI, Go, TypeScript), split by repository into train (20 repositories), validation (5) and test (5). Train labels come from DeepSeek V4.1 Flash (MIT license) following a written labeling guide. Test labels come from two frontier models from different companies, with a model from a third company settling their disagreements. About 18% of hunks are high. Details are in the dataset card.
low answer is not a statement that a change is safe.Apache-2.0, like the base model.
import torchfrom peft import PeftModelfrom transformers import AutoModelForImageTextToText, AutoTokenizer repo = "sarpaydemir/hunkrisk-qwen3.5-2b-lora"THRESHOLD = 0.148 # 0.90 recall on validation; 0.5 follows the model's own answer # Qwen3.5 is multimodal. AutoModelForCausalLM builds a text-only variant with different module# names, and the adapter would silently fail to attach to it.base = AutoModelForImageTextToText.from_pretrained( "Qwen/Qwen3.5-2B", dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(base, repo).eval()tokenizer = AutoTokenizer.from_pretrained(repo) SYSTEM = ( "Classify the diff hunk for code review risk. Reply with one JSON object: " '{"risk": "high" or "low", "categories": [...], "reason": "<one sentence>"}. ' 'Categories: auth, money, schema, data, secrets; use ["none"] when risk is low.')file = "app/Http/Controllers/InvoiceController.php"hunk = """@@ -40,9 +40,6 @@ public function destroy(Request $request, Invoice $invoice)- if ($invoice->user_id !== $request->user()->id) {- abort(403);- } $invoice->delete();""" messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": f"{file}\n{hunk}"}]prompt = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, enable_thinking=False) # P(high): score the risk token right after the answer prefix.high, low = (tokenizer.encode(t, add_special_tokens=False)[0] for t in ("high", "low"))inputs = tokenizer(prompt + '{"risk": "', return_tensors="pt", add_special_tokens=False)with torch.inference_mode(): logits = model(**inputs.to(model.device)).logits[0, -1]p_high = torch.softmax(logits[[high, low]].float(), dim=0)[0].item()print(f"P(high) = {p_high:.3f}, flag = {p_high >= THRESHOLD}") # The full answer, with categories and reason.inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)output = model.generate(**inputs, max_new_tokens=160, do_sample=False)print(tokenizer.decode(output[0, inputs.input_ids.shape[1] :], skip_special_tokens=True))