Test results
Paired change against the untrained Qwen3.5-2B on the FINAL-v3 test sets (400 questions per task; 95%
bootstrap CI over questions; one training seed). For false alarms, negative is better.
Table with columns: Trained on, SOLVE acc, JUDGE step %, JUDGE false alarm, JUDGE Youden J, RECON PC-v4| Trained on | SOLVE acc | JUDGE step % | JUDGE false alarm | JUDGE Youden J | RECON PC-v4 |
|---|
| RECONSTRUCT (step 104) | −10.7 [−13.7, −7.7] | +14.2 [+7.2, +21.8] | +10.2 [+3.9, +16.5] | +7.9 [−1.2, +17.0] | +21.4 [+19.1, +23.5] |
RECONSTRUCT test PC-v4 61.4 (untrained 40.1). It costs 10.7 points of SOLVE accuracy, and its JUDGE gain is mostly response bias: it calls most solutions incorrect (criterion c −0.46), so Youden's J is not significant.
Training
- Algorithm: Dr. GRPO (no reward-std normalisation, constant length normaliser), 8 answers per question, one
update per batch, KL β = 0, token-level importance ratio against vLLM sampling log-probs (capped at 2).
- Reward: the Jev staged rubric score (premise, target, evidence, validity; each 0–4) × 6.25/100, minus 0.1 for several question drafts; shortcuts and gate failures score 0. Truncated or format-gated answers score 0.
- Recipe: lr 5e-5 constant, 8 questions × 8 answers per step; step 104 chosen by best dev Jev score among steps 26/52/78/104. 832 questions seen.
- LoRA: r 64, α 64, dropout 0, all attention and MLP projections (62.5M trainable parameters); AdamW, grad clip
1.0; sampling T 1.0, 16k-token cap; 2 × L40S.
Use
Prompts are the project's v2 student prompts (data/prompts/ in the code); answers are scored after strict
format gates.
vllm serve Qwen/Qwen3.5-2B --enable-lora --max-lora-rank 64 \ --lora-modules reconstruct=Aryama/qwen3.5-2b-grpo-reconstruct
from peft import PeftModelfrom transformers import AutoTokenizer, Qwen3_5ForConditionalGenerationbase = Qwen3_5ForConditionalGeneration.from_pretrained("Qwen/Qwen3.5-2B", dtype="bfloat16")model = PeftModel.from_pretrained(base, "Aryama/qwen3.5-2b-grpo-reconstruct")tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-2B")
Evaluation: T 1.0, top-p 1.0, 16k first pass continued to 32k. Other adapters from the same study: solve · judge · mixed.
Limitations
One seed per arm; the single-task arms use different recipes and amounts of data, so cross-task comparisons mix task
and recipe effects. JUDGE and RECONSTRUCT rubric scores come from a Qwen3.8-27B judge calibrated against DeepSeek
(r 0.94 / 0.84).
Authors
Aryama V Murthy, Mukund Hebbar, Yashas Nataraj Kotre, Anivarth Varma Penumetcha (IIIT Hyderabad, Advanced NLP,
Monsoon 2026). Code and reports: github.com/MukundHebbar/anlp-project.