Test results
Paired change against the untrained Qwen3.5-2B on the FINAL-v3 test sets (400 questions per task; 95%
bootstrap CI over questions; one training seed). For false alarms, negative is better.
Table with columns: Trained on, SOLVE acc, JUDGE step %, JUDGE false alarm, JUDGE Youden J, RECON PC-v4| Trained on | SOLVE acc | JUDGE step % | JUDGE false alarm | JUDGE Youden J | RECON PC-v4 |
|---|
| JUDGE (step 26) | −1.1 [−4.1, +1.9] | +17.2 [+10.2, +24.0] | −24.2 [−29.2, −19.4] | +28.4 [+19.5, +36.9] | +6.0 [+3.3, +8.6] |
JUDGE test verdict accuracy 78.5% (untrained 64.3%), error-finding 52.0% (34.8%), false alarms 13.0% (37.2%). Its checking skill sits in LoRA layers 0–5: copying those layers into the SOLVE adapter adds +15.3 F1 on ProcessBench (Phase 2 of the project).
Training
- Algorithm: Dr. GRPO (no reward-std normalisation, constant length normaliser), 8 answers per question, one
update per batch, KL β = 0, token-level importance ratio against vLLM sampling log-probs (capped at 2).
- Reward: 0 for a wrong verdict; else 0.5 + 0.25 · [first wrong step right] + 0.25 · min(8, E + M)/8, with explanation (E) and soundness (M) levels from an LLM judge (Jev). Truncated or format-gated answers score 0.
- Recipe: lr 5e-5 constant, 8 questions × 8 answers per step; step 26 of 104 chosen by best dev verdict + step accuracy among steps 26/52/78/104. 208 questions seen.
- LoRA: r 64, α 64, dropout 0, all attention and MLP projections (62.5M trainable parameters); AdamW, grad clip
1.0; sampling T 1.0, 16k-token cap; 2 × L40S.
Use
Prompts are the project's v2 student prompts (data/prompts/ in the code); answers are scored after strict
format gates.
vllm serve Qwen/Qwen3.5-2B --enable-lora --max-lora-rank 64 \ --lora-modules judge=Aryama/qwen3.5-2b-grpo-judge
from peft import PeftModelfrom transformers import AutoTokenizer, Qwen3_5ForConditionalGenerationbase = Qwen3_5ForConditionalGeneration.from_pretrained("Qwen/Qwen3.5-2B", dtype="bfloat16")model = PeftModel.from_pretrained(base, "Aryama/qwen3.5-2b-grpo-judge")tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-2B")
Evaluation: T 1.0, top-p 1.0, 16k first pass continued to 32k. Other adapters from the same study: solve · reconstruct · mixed.
Limitations
One seed per arm; the single-task arms use different recipes and amounts of data, so cross-task comparisons mix task
and recipe effects. JUDGE and RECONSTRUCT rubric scores come from a Qwen3.8-27B judge calibrated against DeepSeek
(r 0.94 / 0.84).
Authors
Aryama V Murthy, Mukund Hebbar, Yashas Nataraj Kotre, Anivarth Varma Penumetcha (IIIT Hyderabad, Advanced NLP,
Monsoon 2026). Code and reports: github.com/MukundHebbar/anlp-project.