verl-grpo-medium-qwen3-4b-step50 API & Inference Endpoint

Benchmark (medium, n=100)

Three roles, thinking off everywhere: Qwen3-4B assistant (this model), Qwen/Qwen3.5-9B user simulator, Qwen/Qwen3.5-27B judge. Flags: --user_sim_prompt sim_gap_paper_lazy_full --assistant_max_tokens 4096 --max_new_turns 14.

Table with columns: condition, BLEU, interactivity, tokens (k), MR
condition	BLEU	interactivity	tokens (k)	MR
base `Qwen3-4B` (no proact)	0.4491	0.831	2.935	0.9870
GRPO step 50	0.4524	0.819	2.281	1.0433
GRPO step 100	0.4827	0.801	2.077	1.0761
GRPO step 129	0.4825	0.744	1.944	1.0316
proact prompting (no training)	0.4669	0.825	1.678	1.1236
← this model (step 50)	0.4524	0.819	2.281	1.0433

⚠️ Read before quoting these numbers

Training peaked at step 100 and then regressed. Step 129 bought no BLEU and cost interactivity. Never select a checkpoint on verl's critic/rewards/mean — it rises through step 129, because the reward credits brevity directly while the benchmark judge penalises the interactivity that brevity costs.
Prompting alone still wins on MR (1.1236 for proact vs 1.0761 for step 100) and needs no training. Step 100 wins on BLEU. Say which metric you mean.
Every arm's MR gain is dominated by the token term, not document quality — so MR overstates quality improvements for all of them.
⚠️ 35/100 train/eval prompt overlap with byte-identical reward targets. Because the BLEU reward target is the test reference on those rows, this is label leakage through the reward. See the dataset card.

Training setup

GRPO, train_batch_size=16, rollout.n=8, lr=1e-6, kl_loss_coef=0.001 (low-var KL), 8 GPUs (FSDP, world_size_8), reward = bleu_score (+1) + interactivity (+1) + token_amount (−0.1). Reference run: wandb photon/verlxcollabllm/e1adivlo, val@0 = 1.0618.

Full setup, the self-contained docker image, and the two acceptance tests are documented in studynotes/new_cluster_onboarding.md of the code repo.

Benchmark (medium, n=100)

Table with columns: condition, BLEU, interactivity, tokens (k), MR
condition	BLEU	interactivity	tokens (k)	MR
base `Qwen3-4B` (no proact)	0.4491	0.831	2.935	0.9870
GRPO step 50	0.4524	0.819	2.281	1.0433
GRPO step 100	0.4827	0.801	2.077	1.0761
GRPO step 129	0.4825	0.744	1.944	1.0316
proact prompting (no training)	0.4669	0.825	1.678	1.1236
← this model (step 50)	0.4524	0.819	2.281	1.0433

⚠️ Read before quoting these numbers

Training peaked at step 100 and then regressed. Step 129 bought no BLEU and cost interactivity. Never select a checkpoint on verl's critic/rewards/mean — it rises through step 129, because the reward credits brevity directly while the benchmark judge penalises the interactivity that brevity costs.
Prompting alone still wins on MR (1.1236 for proact vs 1.0761 for step 100) and needs no training. Step 100 wins on BLEU. Say which metric you mean.
Every arm's MR gain is dominated by the token term, not document quality — so MR overstates quality improvements for all of them.
⚠️ 35/100 train/eval prompt overlap with byte-identical reward targets. Because the BLEU reward target is the test reference on those rows, this is label leakage through the reward. See the dataset card.

Training setup

Full setup, the self-contained docker image, and the two acceptance tests are documented in studynotes/new_cluster_onboarding.md of the code repo.

verl-grpo-medium-qwen3-4b-step50

README

Benchmark (medium, n=100)

⚠️ Read before quoting these numbers

Training setup

Explore FriendliAI today

README

Benchmark (medium, n=100)

⚠️ Read before quoting these numbers

Training setup