Recipe
- Base model:
Qwen/Qwen3-VL-4B-Instruct
- Adapter: LoRA, rank 32, alpha 32 (
task_type: CAUSAL_LM)
- Algorithm: GRPO (verl), group size 16, seed 42, ~250 steps
- Max new tokens: 768
- Prompt / reward: taxonomy-traversal ('unlockable' formulation). Shaped OVEN reward (format
0.05, exact/fuzzy 0.70,
specificity-weighted hF 0.15, path-match 0.05, aggregation 0.05); a missing \boxed{}
answer returns 0.00.
Training dynamics (from the wandb run)
Table with columns: metric, start, end, notes| metric | start | end | notes |
|---|
| policy entropy | 0.592 | 0.030 | min 0.030 |
| KL to reference | 0.001 | 8.471 | max 8.471 |
| training reward | 0.427 | 0.362 | — |
| response length | 295.613 | 299.578 | tokens |
|
Training reward moved 0.427 → 0.362 while held-out exact-match went 0.188 → 0.190 (essentially flat). Consistent with the thesis: outcome-only GRPO does not raise validation accuracy above the prompt-only elicitation floor. As a small-KL LoRA update the policy stays close to the reference model.
Usage
from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor
base = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3-VL-4B-Instruct", trust_remote_code=True, device_map="auto")
model = PeftModel.from_pretrained(base, "jucamohedano/qwen3-vl-4b-oven-grpo-traversal")
processor = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-4B-Instruct", trust_remote_code=True)
Intended use
Research artifact for reproducing the thesis's (largely negative) reinforcement-learning results
on OVEN and for studying GRPO training dynamics on a multimodal open-world task. Not tuned or
recommended for production classification.
Provenance
MSc thesis, University of Trento (Juan Camacho Mohedano). Training: verl (branch grpo-oven-v080);
reward in verl/utils/reward_score/oven_boxed.py; data by oven-mllm-eval/scripts/build_verl_oven_parquet.py.
wandb run: offline-run-20260708_034711-qwen3-vl-4b-oven-grpo-trav10-768-unlockable-shaped-n16-seed42.