Result
Same Inspect MBPP evaluation command, seed, sampling, and JSON extraction were used for base and adapter.
Table with columns: Model, Inspect MBPP extracted accuracy, stderr, Eval Job| Model | Inspect MBPP extracted accuracy | stderr | Eval Job |
|---|
Qwen/Qwen3-0.6B | 0.2607003891050584 | 0.027438546123673805 | 6a3564cc3093dba73ce2a4d1 |
Qwen/Qwen3-0.6B + this adapter | 0.3852140077821012 | 0.030415365154196253 | 6a3567c93093dba73ce2a55d |
Absolute gain: +0.1245136186770428.
Note: the temporary evaluation summary script collapsed duplicate Inspect metric names and retained the final MBPP reducer value. The base and adapter were compared with the same extraction protocol.
Training
- Method: TRL
SFTTrainer with completion-only loss and PEFT LoRA.
- Base model:
Qwen/Qwen3-0.6B at commit c1899de289a04d12100db370d81485cdf75e47ca.
- Dataset:
google-research-datasets/mbpp, sanitized config, train split, revision 4bb6404fdc6cacfda99d4ac4205087b89d32030c.
- Selection: trained for 125 steps and loaded the checkpoint with best validation
eval_loss (0.7686 at step 25).
- Train Job:
6a3564dd953ed90bfb944aca.
- Trackio dashboard: .
This adapter is a small MBPP format-recovery experiment, not a general coding model. It should be evaluated on any downstream coding benchmark before use.