SingleBicycle

ccpo-alfworld-qwen2.5-7b-return-hard

README

License: apache-2.0

What this is

An agent policy trained with CCPO (Context-Conditioned Policy Optimization) on ALFWorld, a text-based embodied household benchmark. The base model is Qwen2.5-7B-Instruct.

CCPO targets credit assignment in multi-turn agent RL. Methods in the GRPO family split the advantage into an episode term and a step term. CCPO keeps the episode term untouched and replaces only the step term with a context-conditioned, uncertainty-shrunk leave-one-out baseline, so a head-to-head against the baseline isolates the estimator and nothing else.

The step term works like this. Entries are bucketed by (task_uid, observation_text), the same anchor-state grouping GiGPO and G2PO use. Each entry gets a frozen feature vector from the reference policy's last-prompt-token hidden state, whitened across the batch. Siblings are weighted by exp(-d/tau) on that feature distance, and the baseline is a weighted leave-one-out mean over other trajectories. A shrinkage factor then pulls the result back toward the uniform baseline wherever the disagreement between the two does not exceed its own sampling noise.

Code: tracyhann/agent-context-grpo, built on verl-agent.

This arm

The exact (task, observation) anchor gate GiGPO and G2PO use, with uniform sibling weighting and the step credit computed on the discounted return-to-go.

Flags that distinguish it: gate=hard, target=return, rho=0.0

Results so far

Held-out evaluation on 128 unseen ALFWorld tasks (eval_in_distribution), sampled at temperature 0.4, which is the protocol the published baselines use.

Converged window (step >= 70), the number to read:

Table with columns: metric, value
metric	value
held-out success	83.5%
standard deviation	6.5
evaluations averaged	10

Best single evaluation, step 115:

Table with columns: Held-out, Turns, Pick, Look, Clean, Heat, Cool, Pick2
Held-out	Turns	Pick	Look	Clean	Heat	Cool	Pick2
93.8	12.5	100.0	84.5	96.7	87.5	83.8	96.2

Training success rate at that step was 99.2%. "Turns" is the mean number of environment interactions per held-out episode; it starts near 39 and falls as the policy learns to solve tasks directly rather than by searching.

Do not read the best row as the result. It is the maximum of a noisy series and is biased upward by roughly 1.5 standard deviations. A single 128-episode evaluation here carries about +/-5 points. The converged mean is the honest summary.

The three arms, side by side

Converged-window means, all on identical hardware, seed and data order:

Table with columns: arm, held-out mean, sd, n
arm	held-out mean	sd	n
CCPO return-hard + task backoff + J-weighting	90.0	3.1	9
CCPO return-hard	83.5	6.5	10
CCPO global gate, successor-node target	77.5	6.3	9

What this does and does not show

It does not show that CCPO beats GRPO. No valid GRPO baseline exists in this codebase; an earlier attempt was contaminated when a relaunch passed the run tag but not the estimator. No claim of superiority over GRPO is made here.

These numbers are not comparable to the published ALFWorld results. Those (GRPO 72.8, GiGPO K=2 90.16, HGPO K=2 92.77) are all Qwen2.5-1.5B-Instruct. This is a 7B model, so it is a different setting entirely, and a higher number here says nothing about the estimator relative to those methods.

Every arm here is one seed. The published rows are three. The repository's own guidance is to report the mean of at least three seeds and never to read single points, so differences between the arms above should be treated as unresolved until replicated.

What the comparison between arms does isolate is the estimator: the three runs share hardware, seed, batch and data order, and were verified to produce identical rollouts at step 1 before any gradient was applied.

Training setup

Table

base model	Qwen2.5-7B-Instruct
benchmark	ALFWorld (AlfredTWEnv), 6 task types
batch	16 tasks x 8 rollouts = 128 episodes/step
max env steps	50 per episode, history length 2
learning rate	1e-6, constant
KL	0.01, low-variance estimator
gamma	0.95
max prompt / response	2048 / 512 tokens
eval

Hyperparameters follow the G2PO reference ALFWorld script so the protocol matches the published baselines; only the backbone size and GPU count differ.

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "SingleBicycle/ccpo-alfworld-qwen2.5-7b-return-hard"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

The policy expects the ALFWorld agent prompt used by verl-agent and emits <think>...</think><action>...</action>. The text inside <action> is passed to the environment, so it must match an admissible action.

Status

Interim. This arm is at step 119 of 150. The checkpoint published here is the best-scoring one so far (step 115). Final weights and settled numbers will replace these once the run completes.

Citation

Built on verl-agent (Apache 2.0) and verl. ALFWorld is from alfworld/alfworld.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

SingleBicycle

Model Tree

Base

Qwen/Qwen2.5-7B-Instruct

Fine-tuned

this model

Input Modalities

Text

Output Modalities