Test results
Paired change against the untrained Qwen3.5-2B on the FINAL-v3 test sets (400 questions per task; 95%
bootstrap CI over questions; one training seed). For false alarms, negative is better.
Table with columns: Trained on, SOLVE acc, JUDGE step %, JUDGE false alarm, JUDGE Youden J, RECON PC-v4| Trained on | SOLVE acc | JUDGE step % | JUDGE false alarm | JUDGE Youden J | RECON PC-v4 |
|---|
| Mixed ⅓ each (step 75) | +17.1 [+14.3, +19.9] | +32.2 [+25.9, +38.6] | −29.8 [−34.2, −25.5] | +45.4 [+37.4, +52.6] | +18.3 [+16.1, +20.5] |
SOLVE 69.8%, JUDGE verdict 87.0% (Qwen3.5-9B: 87.6%), false alarms 7.5% (9B: 17.8%), RECONSTRUCT 58.4. Matches the SOLVE specialist (+0.6, n.s.), beats the JUDGE specialist (+15.0 error-finding), is 3.0 below the RECONSTRUCT specialist, and has no negative transfer. It is also the solver and judge used in the project's Phase 3 self-verification study.
Training
- Algorithm: Dr. GRPO (no reward-std normalisation, constant length normaliser), 8 answers per question, one
update per batch, KL β = 0, token-level importance ratio against vLLM sampling log-probs (capped at 2).
- Reward: each task's own reward (SOLVE: verified answer; JUDGE: verdict, step and explanation; RECONSTRUCT: staged rubric). Truncated or format-gated answers score 0.
- Recipe: lr 2e-5 (5-step warm-up), 30 questions × 8 answers per step, every batch exactly 10 + 10 + 10, each task with its own reward, overlapped rollouts; step 75 of 80 chosen by best combined rolling training reward (not dev-checked). 750 questions per task seen.
- LoRA: r 64, α 64, dropout 0, all attention and MLP projections (62.5M trainable parameters); AdamW, grad clip
1.0; sampling T 1.0, 16k-token cap; 2 × L40S.
Use
Prompts are the project's v2 student prompts (data/prompts/ in the code); answers are scored after strict
format gates.
vllm serve Qwen/Qwen3.5-2B --enable-lora --max-lora-rank 64 \ --lora-modules mixed=Aryama/qwen3.5-2b-grpo-mixed
from peft import PeftModelfrom transformers import AutoTokenizer, Qwen3_5ForConditionalGenerationbase = Qwen3_5ForConditionalGeneration.from_pretrained("Qwen/Qwen3.5-2B", dtype="bfloat16")model = PeftModel.from_pretrained(base, "Aryama/qwen3.5-2b-grpo-mixed")tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-2B")
Evaluation: T 1.0, top-p 1.0, 16k first pass continued to 32k. Other adapters from the same study: solve · judge · reconstruct.
Limitations
One seed per arm; the single-task arms use different recipes and amounts of data, so cross-task comparisons mix task
and recipe effects. JUDGE and RECONSTRUCT rubric scores come from a Qwen3.8-27B judge calibrated against DeepSeek
(r 0.94 / 0.84).
Authors
Aryama V Murthy, Mukund Hebbar, Yashas Nataraj Kotre, Anivarth Varma Penumetcha (IIIT Hyderabad, Advanced NLP,
Monsoon 2026). Code and reports: github.com/MukundHebbar/anlp-project.