Highlights
Compared with the base Qwen3.8-27B on 100 functions from repositories it never saw in training:
Table with columns: Base, Maris Piper | Base | Maris Piper |
|---|
| Ranks buggy code above clean code (AUC, 0.5 is chance) | 0.63 | 0.81 |
| Real bugs flagged | 38 of 61 | 43 of 61 |
| Clean functions correctly passed | 23 of 39 | 27 of 39 |
| Flags that name the exact defect later fixed | 2 | 6 |
| Output needed to reach a verdict | up to 4,752 tokens | under 400 tokens |
- Better at telling buggy from clean. It catches more real bugs and raises fewer false alarms at the same time.
- Always gives an answer, and a short one. A few sentences of reasoning, then a clear
Verdict: line. The base model often runs for thousands of tokens before deciding.
- A usable score, not just a yes or no. The probability at the verdict line ranks functions by risk, so a review tool can show the most suspicious first.
- Runs on one GPU. A 435 MB adapter on the 4-bit base, in about 20 GB of memory.
Full results and caveats are under Results.
What it is for
- Finding bugs in code.
- It is the model behind an open source pull request review application.
- Ranking functions by how likely they are to contain a bug using the probability score.
The adapter was trained on this one format. Other formats are untested.
System message:
You are reviewing one function from a pull request. The function is shown first, then the pull request's changes to its file as a unified diff. First reason in a few sentences: what the change did, what the function promises its callers and the rest of the system, and whether some concrete sequence of events breaks that promise. Then write a line reading exactly `Verdict: break` or `Verdict: no break`. After `Verdict: break`, state the break in one to three sentences and quote the code where it goes wrong.
User message: a heading with the file path, the whole function, then the diff of that file in a fenced block.
### src/path/to/file.ts<the function as it is after the change> ### Changes to this file in the pull request```diff<unified diff of the file>```
Thinking mode is off (enable_thinking=False). Answers run to about 95 words; 400 new tokens is enough.
Usage
The adapter was trained and evaluated on the base model loaded in 4-bit. Loading the base in 16-bit is untested.
import torchfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig BASE = "Qwen/Qwen3.8-27B"ADAPTER = "cameronpyne-smith/maris-piper-bug-finder-qwen3.8-27b-lora"SYSTEM = "You are reviewing one function from a pull request. The function is shown first, then the pull request's changes to its file as a unified diff. First reason in a few sentences: what the change did, what the function promises its callers and the rest of the system, and whether some concrete sequence of events breaks that promise. Then write a line reading exactly `Verdict: break` or `Verdict: no break`. After `Verdict: break`, state the break in one to three sentences and quote the code where it goes wrong." quantisation = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True)tokeniser = AutoTokenizer.from_pretrained(BASE)base = AutoModelForCausalLM.from_pretrained(BASE, quantization_config=quantisation, dtype=torch.bfloat16, device_map="cuda")model = PeftModel.from_pretrained(base, ADAPTER).eval() path = "src/example.ts"function = "..."diff = "..."prompt = f"### {path}\n{function}\n\n### Changes to this file in the pull request\n```diff\n{diff}\n```"messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": prompt}]ids = tokeniser.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=False, enable_thinking=False).to("cuda")with torch.no_grad(): output = model.generate(ids, max_new_tokens=400, do_sample=False, pad_token_id=tokeniser.pad_token_id)print(tokeniser.decode(output[0, ids.shape[1]:], skip_special_tokens=True))
Needs a GPU with about 20 GB of memory.
Results
Validation set is 100 functions, 61 bugged, 39 clean. The sample is 1 greedy answer and 4 sampled at temperature 0.7 and the score is the mean of the 5 runs.
Table with columns: Measurement, Base Qwen3.8-27B, With this adapter| Measurement | Base Qwen3.8-27B | With this adapter |
|---|
| Ranking, AUC ↑ | 0.631 | 0.814 |
| Real bugs flagged (of 61) ↑ | 38 | 43 |
| …of those, naming the defect that was later fixed ↑ | 2 | 6 |
| Clean functions passed (of 39) ↑ | 23 | 27 |
| Balanced accuracy ↑ | 0.606 | 0.699 |
| Tokens needed to reach a verdict ↓ | up to 4,752 | under 400 |
↑ higher is better, ↓ lower is better, bold marks the better result.
How to read this:
- Sample size. 100 functions from three repositories is small. Three training runs with the same settings scored AUC 0.78, 0.81 and 0.81; this adapter is the best of the three, so expect about 0.80 on new code. Differences of a few functions are within noise.
- The base model was given more room. It often does not conclude within 400 tokens, so it was given 1,200, and the 28 functions where its greedy answer still had no verdict were rerun at 8,000. In the other 72, 44 of 288 sampled answers still had no verdict at 1,200 tokens and count as undecided (0.5).
- "Clean" is not proven clean. It means no later fix touched the file.
Training data
1,177 examples from merged pull requests in 34 public, permissively licensed repositories, about 58% TypeScript and 42% C#.
759 functions containing a bug that was later fixed, and 418 with no later fix. Each is paired with a short written review ending in a verdict.
Sources include ASP.NET Core, Entity Framework Core, Vue, Vite, NestJS and other widely used C# and TypeScript projects. The dataset is not published.
The 100 validation functions come from three other repositories (zod, Humanizer, Quartz.NET).
Training
Table | |
|---|
| Method | supervised fine-tuning on the answer tokens only |
| LoRA | rank 16, alpha 32, no dropout, on the attention and feed-forward projections |
| Base precision | 4-bit (nf4, double quantisation, bfloat16 compute) |
| Schedule | 1 pass over the data, 166 optimiser steps, peak learning rate 1e-4 with warm-up and cosine decay |
| Validation loss | 1.31 before, 0.87 after (nats per token; lower is better) |
The LoRA layers, optimiser and training loop are written from scratch in couch.
Limits
- C# and TypeScript only; other languages are untested.
- One function and one file's diff at a time. Defects that need code outside the prompt are out of reach.
- Trained on widely used libraries and frameworks; application code may look different.
- Prompts over 16,000 characters were excluded from training.
Versions
- v0.1 (2026-11-15): first release.