GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
idealab-cs2
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
idealab-cs2
Model Tree
Qwen/Qwen3-4B-Thinking-2507
Input Modalities
Output Modalities
Supported Functionality
mean(z) − 0.5·std(z), a
disagreement guard (Coste et al. 2023) that down-weights answers the two scorers disagree on.Reappraisals were compared pairwise against GPT-4-0314's reappraisals for the same scenarios, judged by Llama-3.1-70B-Instruct scoring both A/B orderings. That judge was selected by calibration against the human preference data (93% agreement on clearly separated pairs; larger judges such as Llama-3.1-405B-FP8 and GLM-5.2 agreed less and showed position bias). Its agreement on very close comparisons, like the ones below, is lower. Results are pooled over three independent evals of this checkpoint (fresh generations + an independent both-orders judge each: 0.883, 0.879, 0.854), n = 720.
| model | win-rate vs GPT-4-0314 | n | 95% CI lower (iid, optimistic) |
|---|---|---|---|
| Reappraisal-4B-GRPO-Committee (this model) | 0.872 | 720 | 0.846 |
| Reappraisal-4B-GRPO-RMv2 (single reward model) | 0.806 | 720 | 0.776 |
| Reappraisal-4B-GRPO (v1, BT reward) | 0.581 | 480 | 0.537 |
| DeepSeek-R1-671B (single-pass reference) | 0.771 | 240 | 0.718 |
Read the CI column as a screening tool, not a precise bound. The 720 contests cluster on only 6 vignettes (n_effective ≈ 6), so the iid Wilson CIs above are optimistic. The honest vignette-cluster bootstrap CI for this model's headline is [0.781, 0.948] — CI-low 0.781, still far above the 0.608 project bar and the 0.5 tie line, so the win over GPT-4-0314 on these 6 vignettes survives clustering. But the CI-low is 0.781, not 0.846 (retracted). This is a specialist trained on these 6 scenarios against zero-shot generalists, so read it as evidence about reward-model quality on a narrow task, not a general capability ranking.
On DeepSeek-R1, be precise about what holds. Comparing our cluster CI-low to R1's reference point (0.771) is not a valid test — that reference-rate comparison is retracted. What does hold is the head-to-head on the 6 vignettes: this model's reappraisals vs R1-671B's, same scenarios, both orderings, same judge, wins 0.6729 (n = 240), vignette-cluster CI [0.550, 0.804], CI-low 0.550 > 0.5 — a 4B single-pass model out-reappraising a 671B reasoning model pairwise, in distribution.
Out-of-distribution: we do NOT beat R1, and this model is at parity off-distribution. The policy is trained on the same 6 vignettes it is evaluated on, so we tested fresh scenarios it never trained on (n-gram disjoint). On a fresh cross-family pool this model is at parity-or-below vs a strong open 72B (Qwen2.5-72B): 0.4875, cluster CI [0.398, 0.579] — the earlier "+6.8pt over the single-reward policy" claim is retracted (the honest like-for-like gap is within noise). And against DeepSeek-R1-671B off-distribution our 4B loses: single-pass ~0.39–0.45 (cluster CI-low < 0.5), and best-of-N test-time compute only tied on the arm-selection pool — a pre-registered confirm on a fresh held-out pool reverted to 0.375 / 0.378 on both seeds (winner's curse). Six+ approach families were adversarially audited and each came back negative. This is a genuine capability ceiling: a 4B cannot out-reappraise R1-671B on fresh scenarios; off-distribution generalization tracks base capacity, not our training.
Why the committee helps. The gain comes from the heterogeneity, not from simply adding a second reward model. An RM-heavy version of the same committee (weights 0.7 / 0.3 toward the discriminative model) scored only 0.760, below the single-reward RMv2 recipe, because down-weighting the generative model's independent signal collapses the committee back toward a single-reward regime. Weighting the two mechanisms equally is what buys the improvement: the two scorers make different mistakes, and rewarding only their agreement penalizes the answers that game either one alone. (Using either generative reward alone as the training signal collapses the policy; the committee, not the generative model, is what works.)
The model was trained with this system prompt:
text
You are helping someone reduce a negative emotion they feel in a short interpersonal scenario byoffering an alternative interpretation of the situation (a 'reappraisal' / 'rethinking'). Directyour response at the person in the scenario in second person. Limit your response to two sentencesmaximum. Do not list emotions; output only the reappraisal text.The user turn is
text
SCENARIO:\n{scenario}\n\nWrite a reappraisal of this scenario in two sentences maximum, addressing the person in second person.<think>...</think> followed by the reappraisal; only the text after
</think> is the answer.
The reward carries a format gate penalizing outputs that are empty or longer than two sentences (so the policy cannot inflate reward through verbosity). Training was healthy: reward rising, KL bounded (~0.60), entropy alive, no reward-std collapse.
The 6 vignettes' GPT-4-0314 reappraisals and the frontier-opponent answers are held out and used only for evaluation; training uses the vignette prompts (the task) and the human-rating-based reward models. Both committee scorers are Qwen-family, distinct from the Llama-3.1-70B evaluation judge, and the generative model's references are real human reappraisals, never the held-out GPT-4 answers or the judge's outputs. Any pooled or synthetic training text is n-gram-checked against the held-out eval answers before use.
Research artifact for the study of RLHF, reward modeling, and computational emotion regulation. It writes short cognitive reappraisals of negative interpersonal situations. It is not a clinical or mental-health tool and must not be used as one. Quality is validated only on the narrow reappraisal task and the evaluation above.
seed0..2/ discriminative ensemble + grm/ generative member)You are helping someone reduce a negative emotion they feel in a short interpersonal scenario byoffering an alternative interpretation of the situation (a 'reappraisal' / 'rethinking'). Directyour response at the person in the scenario in second person. Limit your response to two sentencesmaximum. Do not list emotions; output only the reappraisal text.