GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
yuyu0529nya
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
yuyu0529nya
Model Tree
Qwen/Qwen2.5-7B-Instruct
Input Modalities
Output Modalities
Supported Functionality
GRPO on τ²-bench-airline from raw Qwen2.5-7B-Instruct is null — and not because of a bug.
Base success is ~20%, so most rollout groups come back all-fail → zero intra-group variance → the
group-relative advantage is ~0 → no gradient. RL sharpens what a policy already does sometimes;
it cannot invent a skill. Worse, the base policy's own successful rollouts never invoke the
write-tools (update_reservation_*) that held-out tasks require, so self-sampling can't bootstrap
them either.
This adapter is the floor that makes the RL step work.
This adapter, merged into the base, is the step 0 policy of the
tau2_airline verl recipe. Measured under
that recipe's protocol (held-out BINARY mean@4, 20 held-out airline tasks, VAL_TEMP=0.5,
llama-3.3-70b user simulator):
| val@0 (this adapter) | val@20 (after GRPO) | |
|---|---|---|
| seed 42 | 0.275 | 0.5625 |
| seed 123 | 0.375 | 0.55 |
GRPO endpoint across 2 seeds: 0.556 ± 0.01. Measured evaluation noise on this same checkpoint (6 repeats of val@0) is 0.30 ± 0.05, so the two val@0 values above are a single checkpoint sampled twice from that band, not a difference.
Scope note. A separate hand-written pipeline reported
for this approach, but under a different eval protocol (pass^1 over 10 trials with a self-hosted 7B user simulator, n=200). Those numbers are not directly comparable to the table above — different user simulator, different metric — and I have not re-verified that this exact file is the checkpoint they were measured on. The table above is what this file scores.text
base 0.20 → distilled SFT 0.405 → GRPO 0.545
python
from peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct", torch_dtype="auto", device_map="auto")model = PeftModel.from_pretrained(base, "yuyu0529nya/qwen2.5-7b-tau2-airline-sft-lora")tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")To use it as an RL starting point, merge first (model.merge_and_unload()) and point the trainer at
the merged weights — that is exactly how the recipe above consumes it.
This adapter only makes sense inside the τ²-bench airline harness — same system prompt, same 14 tool schemas, same dialogue loop. It is a policy for that environment, not a general chat model.
| base | Qwen/Qwen2.5-7B-Instruct |
| method | behavior-cloning SFT (QLoRA 4-bit) on teacher trajectories |
| teacher | DeepSeek V4 Flash (deepseek-v4-flash, requested via the deepseek-chat alias) — agent and user-simulator, temp 0.7 |
| data | yuyu0529nya/tau2-airline-deepseek-distill (ds_114 config) — 114 successful teacher trajectories (reward = 1), deduped by action signature, capped 4/task |
| LoRA |
Teacher success on these tasks was ~76% vs base ~20%, and 51% of teacher trajectories invoke the write-tools the base policy never reaches on its own. That skill gap is why self-RFT from base cannot work here, and why distillation — rather than more RL — was the fix.
Qwen2.5-7B-Instruct, Apache-2.0 → derivatives unrestricted.@example.com, RFC 2606 reserved). No real PII.llama-3.3-70b. Not controlled for.airline domain only. Not evaluated on other τ²-bench domains. Not a general-purpose model.r=16, alpha=32, dropout=0.05, targets = q,k,v,o,gate,up,down_proj |
| loss | assistant-token-only (render-twice-diff mask) |