Standalone evaluation at this step
128 ALFWorld valid_seen tasks, 3 seeds, temperature 0.4, max 50 env steps:
Table with columns: value | value |
|---|
| success rate | 62.24% ± 3.27 |
base Qwen2.5-1.5B-Instruct | 2.86% ± 0.37 |
| mean episode length | 27.6 (base: 49.1) |
| format rate | 97.8% (base: 98.1%) |
Training curve
In-training validation success rate (val/success_rate), measured every 5 optimizer steps
on a 32-task validation batch (shown every 10 steps below). This is a smaller and noisier
probe than the 128-task standalone evaluation.
Table with columns: optimizer step, success rate (%)| optimizer step | success rate (%) |
|---|
| 0 | 6.2 |
| 10 | 7.8 |
| 20 | 13.3 |
| 30 | 14.8 |
| 40 | 16.4 |
| 50 | 34.4 |
| 60 | 35.2 |
| 70 | 36.7 |
| 80 | 43.0 |
|
Run history
This run (ppo_qwen2.5_1.5b_v3) is the third PPO attempt. An earlier attempt collapsed —
its step-130 checkpoint scored 9.90% with a 1.3% action format rate, i.e. the policy
stopped emitting parseable actions. The run published here was restarted from checkpoints
twice (at step 30 and step 80) for operational reasons, not because of divergence; the
validation curve is continuous across those boundaries.
Hyperparameters
Table with columns: group, parameter, value| group | parameter | value |
|---|
| RL | adv_estimator | gae |
| RL | actor.use_kl_loss / kl_loss_coef | True / 0.01 |
| RL | kl_loss_type | low_var_kl |
| RL | invalid action penalty | True, coef 0.1 |
| Optim |
Files
Table with columns: path, contents| path | contents |
|---|
model.safetensors, config.json, tokenizer files | bf16 weights, merged from the FSDP shards with scripts/model_merger.py. lm_head is omitted because tie_word_embeddings=true (same layout as the base Qwen release). |
training_state/actor/*.pt | FSDP-sharded actor weights, Adam optimizer state, RNG/scheduler state |
training_state/ lets you resume RL training from this exact optimizer step. The shards are
written for world_size=2; resuming on a different number of GPUs requires resharding.
Other checkpoints from this run
step 10 · step 20 · step 30 · step 60 · step 70 · step 80 · step 130 · step 140 · step 150
Not every step was retained: the run used trainer.max_actor_ckpt_to_keep, so some
intermediate checkpoints were pruned during training.
Caveats
- Do not use
is_correct / pass@1 from verl logs; they are hardcoded to 1.0.
Use val/success_rate.
- Evaluated on ALFWorld
valid_seen only; valid_unseen was not run.
- A StraTA reproduction is published alongside these models
(
chanyoungkim/strata-qwen2.5-1.5b-alfworld-step*). It uses a different agent scaffold
(140 eval tasks, temperature 0.7, 7168-token prompts, strategy-conditioned prompting)
and is therefore not directly comparable to the numbers above.
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "chanyoungkim/ppo-qwen2.5-1.5b-alfworld-step140"
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")
tok = AutoTokenizer.from_pretrained(repo)