Recipe
- Code: prime-rl commit
21814b401.
- Config:
examples/basic/reverse-text/rl.toml at that commit with model.name = the SFT checkpoint and orchestrator.renderer.name = "qwen3.5": 20 steps, 8 prompts x 16 rollouts (batch 128), 128 max tokens, lr 3e-6, 1 inference + 1 trainer H200, NCCL weight broadcast.
Numbers
- Train reward per step: 0.30, 0.30, 0.30, 0.32, 0.36, 0.33, 0.35, 0.34, 0.35, 0.35, 0.37, 0.40, 0.37, 0.36, 0.39, 0.38, 0.41, 0.43, 0.41, 0.42.
reverse-text eval reward (256 prompts, temperature 1): step 0: 0.3086, step 5: 0.3543, step 10: 0.3967, step 15: 0.4027, step 20: 0.4077.
Caveat
On Qwen3.5-0.8B, prime-rl RL currently shows a much higher trainer/inference mismatch KL (0.01-0.1 per step) than on Qwen3-0.6B (about 0.002) with the same recipe, and learning stalls after about 20 steps. This is under investigation.