qwen3-4b-symbolic-vanilla-ppo-ent001-v1 API & Inference Endpoint

The result: collapse delayed, then terminal

Table with columns: step, 10, 30, 50, 70, 90, 100-150
step	10	30	50	70	90	100-150
val reward	0.528	0.694	0.778	0.708	0.833	0.000

The no-entropy arm collapsed at step ~80 and self-recovered to 0.972 by 150. This arm survived that window (0.833 at step 90 — its peak, and better than arm 1 ever was pre-collapse), then collapsed at ~100 with entropy pinned at 1e-4 and never recovered (1-turn degenerate episodes through 150). Vanilla PPO on this task lives on a knife's edge around steps 80-120; a 0.01 entropy bonus shifts the edge, it does not remove it.

Files

root: step_90 policy — the pre-collapse peak (val 0.833), value head stripped, vLLM-loadable; value_head.safetensors alongside
checkpoints/step_120/, checkpoints/step_150/ — mid/post-collapse policies for collapse forensics (val 0.000, 1-turn degenerate greedy mode)

Training setup

Same as arm 1: prime-rl ppo, GAE gamma=1.0 lambda=0.95, clip 0.2, value coef 0.5, LR 1e-6, batch 64 x group 8, 150 steps, temp 0.7, symbolic-curriculum-v2 (588 train / 72 val pass@4-mixed tasks), 4x RTX PRO 6000 Blackwell. W&B: https://wandb.ai/krishnapg2315/blog-rl/runs/fc57f9265a2d4ff8aaa80fabcc9058ab

The result: collapse delayed, then terminal

Table with columns: step, 10, 30, 50, 70, 90, 100-150
step	10	30	50	70	90	100-150
val reward	0.528	0.694	0.778	0.708	0.833	0.000

Files

root: step_90 policy — the pre-collapse peak (val 0.833), value head stripped, vLLM-loadable; value_head.safetensors alongside
checkpoints/step_120/, checkpoints/step_150/ — mid/post-collapse policies for collapse forensics (val 0.000, 1-turn degenerate greedy mode)

qwen3-4b-symbolic-vanilla-ppo-ent001-v1

README

The result: collapse delayed, then terminal

Files

Training setup

Explore FriendliAI today

README

The result: collapse delayed, then terminal

Files

Training setup