Training curve
Validation success rate on the full ALFWorld valid_seen split (140 tasks),
measured every 10 optimizer steps during training. Step 0 is the untrained base model.
Table with columns: optimizer step, success rate (%)| optimizer step | success rate (%) |
|---|
| 0 | 0.71 |
| 10 | 0.71 |
| 20 | 3.57 |
| 30 | 1.43 |
| 40 | 14.29 |
| 50 | 18.57 |
| 60 | 6.43 |
| 70 | 22.86 |
| 80 | 37.14 |
| 90 | 42.14 |
| 100 | 43.57 |
| 110 | 50.71 |
| 120 | 69.29 |
| 130 | 85.00 |
| 140 | 83.57 |
| 150 | 86.43 |
Final evaluation (step 150, 2 seeds)
Run with evaluate-strata-alfworld-text-qwen2.5-1.5b.sh: val_only, 140 tasks,
temperature 0.7 / top_p 0.8 / top_k 20, max 50 env steps.
Table with columns: subtask, seed 1, seed 2, mean ± std| subtask | seed 1 | seed 2 | mean ± std |
|---|
| pick | 97.14 | 97.14 | 97.14 ± 0.00 |
| clean | 92.59 | 92.59 | 92.59 ± 0.00 |
| pick2 | 91.67 | 87.50 | 89.58 ± 2.95 |
| heat | 81.25 | 81.25 | |
Auxiliary metrics: action format_rate 1.000 (both seeds), mean episode length
13.2–14.2 steps (base model: 49.7).
Hyperparameters
All algorithm hyperparameters follow the paper: 150 steps, batch size 16, 4 strategies
per task, 8 rollouts per strategy, oversampling ratio sigma=8, aggregation ratio
delta=0.5, length penalty threshold lambda=0.5, self-judgment reward weight kappa=0.1.
Table with columns: group, parameter, value| group | parameter | value |
|---|
| StraTA | num_stras_per_run (strategies per task) | 4 |
| StraTA | num_trajs_per_stra (rollouts per strategy) | 8 |
| StraTA | num_cands_per_stra (oversampling ratio, sigma) | 8 |
| StraTA | delta_coef (aggregation ratio) | 0.5 |
Infrastructure deltas
This run reproduces the upstream 7B recipe on 2x A100 80GB. A full diff of the
resolved Hydra config against train-strata-alfworld-text-qwen2.5-7b.sh shows
6 differing keys out of 533, all infrastructure -- no algorithm or optimization
parameter was changed:
Table with columns: key, upstream 7B, this run, effect on results| key | upstream 7B | this run | effect on results |
|---|
trainer.n_gpus_per_node | 8 | 2 | none -- verl computes ppo_mini_batch_size *= rollout.n then //= world_size, so the global mini-batch is invariant |
actor.fsdp_config.param_offload | True | False | none -- CPU offload only |
actor.fsdp_config.optimizer_offload | True |
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "chanyoungkim/strata-qwen2.5-1.5b-alfworld-step040"
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")
tok = AutoTokenizer.from_pretrained(repo)
The model expects the StraTA ALFWorld prompt format: it first emits a plan inside
<strategy>...</strategy>, then acts, emitting one action per turn inside
<action>...</action>.
Files
Table with columns: path, contents| path | contents |
|---|
model.safetensors, config.json, tokenizer files | bf16 weights, ready for from_pretrained. lm_head is omitted because tie_word_embeddings=true (same layout as the base Qwen release). |
training_state/model_world_size_2_rank_{0,1}.pt | FSDP-sharded fp32 weights |
training_state/optim_world_size_2_rank_{0,1}.pt | Adam optimizer state |
training_state/extra_state_world_size_2_rank_{0,1}.pt |
training_state/ lets you resume RL training from this exact optimizer step. Note the
shards are written for world_size=2 (2 GPUs, FSDP). Resuming on a different number
of GPUs requires resharding. To resume, place the files back as
checkpoints/<exp>/models/global_step_40/actor/ and point verl at that directory.
The published model.safetensors is a bf16 cast of the fp32 master weights; the
original fp32 tensors are preserved inside training_state/.
All checkpoints from this run
step 10 · step 20 · step 30 · step 40 · step 50 · step 60 · step 70 · step 80 · step 90 · step 100 · step 110 · · · ·
Caveats
- Do not use
is_correct or pass@1 from the logs. Both are hardcoded to 1.0
in the workflow and are meaningless. The correct metric is
val/average/metrics/success_rate.
- Evaluation uses the StraTA inference protocol (one strategy, then one trajectory
conditioned on it; no diversity selection, no self-judgment). This is what
strata_workflow.run() does when rollout_engine.validate is set, and it matches
evaluate-strata-*.sh.
- Numbers here are not directly comparable to models evaluated under a different
agent scaffold (different prompt format, context window, or sampling temperature).
- The evaluation set is ALFWorld
valid_seen (140 tasks, all of them -- not a
subsample). valid_unseen was not evaluated.
Citation
@article{xue2026strata,
title={StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstraction},
author={Xue, Xiangyuan and Zhou, Yifan and Wang, Zidong and Tang, Shengji and Torr, Philip and Ouyang, Wanli and Bai, Lei and Yin, Zhenfei},
journal={arXiv preprint arXiv:2605.06642},
year={2026}
}