mrzhao13

qwen3.5-2b-shopsimulator-grpo-rl500-1ep

README

License: apache-2.0

Training

Table with columns: Item, Value
Item	Value
Original base model	`Qwen/Qwen3.5-2B`
RL initialization	`mrzhao13/qwen3.5-2b-shopsimulator-sft-512-1ep`
RL task set	`rl_500`, 500 unique tasks
Epochs	1
Candidates per task	4
Candidates / optimizer steps	2,000 / 100
Rollout batch size / global batch size	5 / 20
Learning rate	`1e-6`
Advantage estimator	GRPO
KL loss coefficient / type	`0.001` / `low_var_kl`
Maximum model turns	40
Maximum context / response length	16,384 / 2,048
Maximum tokens per GPU	12,288
Non-zero reward-variance groups	352 / 500 (70.4%)
Completed / truncated / aborted candidates	1,881 / 119 / 0
Training framework	Slime + Megatron-LM + SGLang + Pi
Hardware	One NVIDIA Pro 6000D 84 GB
Weight identity SHA-256	`8eff796683fa32bf74af7378a64646eea190e71cb52e397de431e7da72463f18`

Evaluation

All three checkpoints were evaluated with the same modified ShopSimulator service, deterministic price generation, fixed official_test_200 task slice, and one rollout per task. These are k=1 point estimates.

Table with columns: Model, Positive-reward pass@1, Strict-success pass@1, mean@1 r_loose, mean@1 r_hard
Model	Positive-reward pass@1	Strict-success pass@1	mean@1 `r_loose`	mean@1 `r_hard`
Qwen3.5-2B	2.0%	0.0%	0.004286	0.000000
SFT-512 1 epoch	72.5%	10.5%	0.389829	0.124417
This GRPO checkpoint	90.5%	31.0%

The RL checkpoint also achieved a 91.5% environment completion rate and a 40.0% right-product pass@1 rate on this fixed evaluation.

Usage

Use this repository as a drop-in replacement for Qwen/Qwen3.5-2B with a recent Transformers version that supports Qwen3.5. Refer to the official Qwen3.5-2B model card for loading and inference examples, replacing the model ID with this repository.

ShopSimulator evaluation requires the companion Pi extension, prompt template, patched environment and deterministic pricing behavior from the pi-slime-shopsimulator source repository when it is published.

Limitations

Optimized only for Chinese ShopSimulator shopping tasks.
Evaluated only in the patched text-based ShopSimulator environment.
The deterministic pricing patch changes scores relative to the unpatched upstream environment.
The official evaluation contains 200 tasks with one stochastic rollout each.
No visual, broad instruction-following or safety evaluation was performed.
Online RL can amplify environment- and reward-specific behavior.
The SFT teacher data contains AI-generated content and may contain errors.

License and attribution

The original Qwen model, the SFT initialization and these derivative weights are distributed under Apache-2.0. The original Qwen license is included in LICENSE. This model card identifies the checkpoint as modified and does not grant rights to third-party ShopSimulator task or environment content.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

mrzhao13

Model Tree

Base

mrzhao13/qwen3.5-2b-shopsimulator-sft-512-1ep

Fine-tuned

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality