GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
SeanWang0027
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
SeanWang0027
Model Tree
Qwen/Qwen3-1.7B
Input Modalities
Output Modalities
Supported Functionality
ScienceWorld test split (200 task variations, AgentGym protocol), 4 independent passes, temperature 0.4, 512 tokens per turn, at most 30 rounds, thinking off, no system turn. Success = final score 100. Mean ± std across the 4 passes.
| model | success rate | Avg@1 (mean final score / 100) |
|---|---|---|
| Qwen3-1.7B (base) | 0.12% ± 0.22 | -0.0331 |
| SFT on gpt-5.4-mini trajectories, 3 epochs | 12.88% ± 1.24 | 0.1557 |
| online ROSE 10+5 with a gpt-5.4-mini teacher, 3 epochs | 17.50% ± 2.29 | 0.2942 |
ROSE minus SFT, paired bootstrap over tasks: +4.62 points success [0.75, 8.75], +0.139 Avg@1 [0.072, 0.203].
A separate 1-epoch run of the same setup scored 8.88%. On the training rollouts the teacher raised the episode score by +0.112 on average over its 5 turns.
The teacher sometimes copies the prompt's action template open/close OBJ literally (the simulator rejects it); this model emits it
on 35.22% of its turns, against 74.88% for the SFT arm.
ReAct. The conversation is AgentGym's ScienceWorld instruction as a user turn, a canned assistant acknowledgement, then one user
turn per observation. The model replies Thought:\n...\n\nAction:\n<one command>. Render with Qwen3's chat template with
enable_thinking=False.
python
from transformers import AutoModelForCausalLM, AutoTokenizertok = AutoTokenizer.from_pretrained(REPO); model = AutoModelForCausalLM.from_pretrained(REPO, torch_dtype="bfloat16")ids = tok.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False, return_tensors="pt")Base Qwen3-1.7B scores 0.12%. Every number is the mean ± std of 4 independent passes at temperature 0.4, 512 tokens per turn, at most 30 rounds, thinking off, no system turn; success = final score 100, Avg@1 = mean final score / 100.
| method | epoch | success rate (avg of 4 passes) | Avg@1 | share of turns that are the literal open/close … | model |
|---|---|---|---|---|---|
| SFT | 1 | not evaluated | — | — | qwen3-1.7b-sciworld-sft-gpt54mini-ep1 |
| SFT | 2 | 13.25% ± 2.56 | 0.1574 | 74.20% | qwen3-1.7b-sciworld-sft-gpt54mini-ep2 |
| SFT | 3 | 12.88% ± 1.24 | 0.1557 | 74.88% | qwen3-1.7b-sciworld-sft-gpt54mini-3ep |
| ROSE 10+5 | 1 | not evaluated | — | — | qwen3-1.7b-sciworld-rose-s10t5-gpt54mini-ep1 |
| ROSE 10+5 | 2 | 7.75% ± 1.92 | 0.1455 | 62.68% | qwen3-1.7b-sciworld-rose-s10t5-gpt54mini-ep2 |
| ROSE 10+5 | 3 | 17.50% ± 2.29 (re-run: 15.38% ± 1.43) | 0.2942 (0.2512) | 35.22% | qwen3-1.7b-sciworld-rose-s10t5-gpt54mini-3ep ← this model |
| ROSE 10+5 | 4 | 20.25% ± 2.25 | 0.3433 | 17.98% | qwen3-1.7b-sciworld-rose-s10t5-gpt54mini-ep4 |
| ROSE 10+5 | 5 | 25.75% ± 1.30 | 0.3692 | 2.77% | qwen3-1.7b-sciworld-rose-s10t5-gpt54mini-ep5 |
| ROSE 10+5 | 6 | 20.75% ± 1.82 | 0.2946 | 26.59% | qwen3-1.7b-sciworld-rose-s10t5-gpt54mini-ep6 |
How to read this table:
gpt-5.4-mini, medium) copies the prompt's action template open/close OBJ literally in 29.89% of its own turns; the
simulator rejects it every time. Students inherit the habit, and their test success tracks how strong the habit is in the
particular checkpoint (last column). In online ROSE the habit oscillates during training, so single-checkpoint numbers are
noisy: ROSE epoch 2 was saved at a relapse peak, epochs 3 and 5 near minima, epoch 6 during another relapse.from transformers import AutoModelForCausalLM, AutoTokenizertok = AutoTokenizer.from_pretrained(REPO); model = AutoModelForCausalLM.from_pretrained(REPO, torch_dtype="bfloat16")ids = tok.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False, return_tensors="pt")