YuyangZhao

skill1-webshop-skill-only-qwen3.5-4b-step40

README

License: apache-2.0

Training summary

Table

Base model	`Qwen/Qwen3.5-4B`, chat-template thinking disabled (`enable_thinking=False`)
Algorithm	GRPO, group size 8, 16 tasks per step, lr 1e-6, KL loss coef 0.01
Trainable phases	retrieval-query generation + skill re-rank. Executor `play` steps excluded from the training batch
Rewards	task outcome, discounted back into the selection steps; re-rank NDCG x 1.0; distillation intrinsic reward x 1.0 (verl defaults; the WebShop launch script passes no lambda overrides)
Skill library	top-k 3 retrieval, utility EMA alpha 0.05, admission requires a binary task success; 19 skills at step 40
Episode length	`env.max_steps=15`, invalid-action penalty 0.5
Steps	40 (checkpoint every 10 steps), 10 h 01 m wall clock
Hardware	4 x NVIDIA L40S (46 GB), vLLM 0.19.1 rollouts, FSDP actor with parameter/optimizer offload

Results

64 WebShop goals per validation pass, temperature 0.4.

Table with columns: step, success rate, with retrieval, success rate, no retrieval, reward (val/text/test_score), skills in library
step	success rate, with retrieval	success rate, no retrieval	reward (`val/text/test_score`)	skills in library
0 (base)	0.125	0.078	0.742	0
10	0.047	0.062	0.269	5
20	0.031	0.141	0.171

Read these numbers with care:

The 64 goals are re-sampled at every validation pass from the held-out split (goal indices 0-499; training draws from 500-6909), so the rows are not scored on a common task set. A difference of one to five successes out of 64 is not meaningful.
With-retrieval success is below the step-0 baseline at every evaluated step. The no-retrieval column moves too, even though the executor receives no gradient, which is consistent with sampling noise dominating at this scale.
val/text/test_score is the mean shaped reward, not a success rate. WebShop also reports a partial-credit metric whose key literally contains a space,
markdown
```
webshop_task_score (not success_rate)
```
; it is not quoted here to avoid confusion.

Why the skill mechanism never engaged

Two measurements from the run explain the flat/declining with-retrieval numbers:

Re-ranking produced no signal at all. bigen_train/rerank_parse/success was 0 out of 128 attempts at every step, and rerank_ndcg was 0.000 throughout training and validation. The policy answered <rank>none</rank> every time, so the NDCG reward term contributed nothing to the gradient.
The skill library stayed tiny. Distillation admits a skill only from a trajectory that actually succeeded (binary success), and WebShop binary success ran at 0-13% per step, so the library grew to only 19 entries in 40 steps. Retrieval therefore returned mostly irrelevant candidates.

In effect the two trainable phases were optimised almost entirely by the discounted task return, with no usable intrinsic signal. On ALFWorld, where binary success is far more common, the same code grew a 1865-entry library over the same number of steps.

Differences from the ALFWorld run

Table with columns: ALFWorld checkpoint, this WebShop checkpoint
	ALFWorld checkpoint	this WebShop checkpoint
`lambda_rerank` / `lambda_distill`	0.3 / 0.3 (explicit overrides)	1.0 / 1.0 (verl defaults; no overrides passed)
`env.max_steps`	50	15
invalid-action penalty	0.1	0.5
library at step 40	1865 skills	19 skills

Environment (important for comparability)

These numbers are not comparable to published WebShop results:

The catalog is a locally cut 1000-product subset: the first 1000 entries of items_shuffle.json with items_ins_v2.json filtered to those ASINs, cut on 2026-09-16. It is not the WebShop authors' released items_shuffle_1000.json. It is selected by the repo default env.webshop.use_small=True, with human_goals=False.
Search is served by the environment's rank_bm25 BM25Okapi fallback, not by the pyserini/Lucene index the original WebShop uses, because pyserini is not installed in this environment.

Usage

Standard Qwen3_5ForConditionalGeneration; loads with transformers >= 5 or vLLM. Trained with the chat template's thinking mode off:

python
from transformers import AutoTokenizer, AutoModelForImageTextToText

repo = "YuyangZhao/skill1-webshop-skill-only-qwen3.5-4b-step40"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForImageTextToText.from_pretrained(repo, dtype="bfloat16", device_map="auto")

messages = [{"role": "user", "content": "..."}]
prompt = tok.apply_chat_template(messages, tokenize=False,
                                 add_generation_prompt=True, enable_thinking=False)

The WebShop prompts expect an action inside <action> </action>; the skill-selection prompts expect <query>...</query> and <rank>...</rank>.

Notes and limitations

Weights are bf16 in a single unsharded model.safetensors.
The base model's 15 mtp.* tensors (multi-token-prediction head) are not included in the merged checkpoint, although config.json still advertises mtp_num_hidden_layers: 1. MTP speculative decoding is therefore unsupported here.
generation_config.json is synthesised from the model config by the merge step, not inherited from the base repo (the local base snapshot ships none), so it carries library defaults rather than Qwen's recommended sampling settings.
The vision tower of the base model is carried through unchanged and unused.
The skill library itself is not part of this repository.
Training run: wandb.ai/yuyangzhao-ustc/skill1_webshop/runs/49f1o5r3.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

YuyangZhao

Model Tree

Base

Qwen/Qwen3.5-4B

Fine-tuned

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality