while-ai

paper-zero-rl-format-reward-4b

README

License: apache-2.0

Result

Run 2026-09-18, both arms, one H100, 46.2 GPU minutes.

Table with columns: Arm, pass@1, 95% CI, pass@4, Steps, GPU min
Arm	pass@1	95% CI	pass@4	Steps	GPU min
Base, no training	0.51	[0.45, 0.57]	0.74	0	0
Baseline (strict boxed reward, -1 without a box)	0.63	[0.57, 0.68]	0.80	30	26.4
Recipe (correctness only)	0.72	[0.66, 0.77]	0.88	30	19.8

Recipe vs baseline: +0.094 [+0.052, +0.139] over 160 paired tasks. Both arms beat the base; the gap between them is the paper's finding and it clears the noise band the three base re-runs set. On the baseline's own strict reward the recipe arm scores lower (0.53 vs 0.58): it boxes less and is right more.

The six seed folders are the 2026-09-21 follow-up the README's Learned section asks for: three training seeds per arm at the same settings. Their paired read-out is not in the recipe README yet; the adapters are here so the numbers can be recomputed.

Arms in this repo

The root holds the arm the recipe README's headline number reports. Every other arm is a subfolder named after it. checkpoints/ never ships.

Table with columns: folder, arm
folder	arm
`.`	recipe arm: correctness-only reward, 2026-09-18 run (the README's table)
`baseline`	baseline arm: strict boxed reward, 2026-09-18 run
`recipe-seed17`	recipe arm, seed 17, 2026-09-21
`recipe-seed18`	recipe arm, seed 18, 2026-09-21
`recipe-seed19`	recipe arm, seed 19, 2026-09-21
`baseline-seed17`

Load

python

from peft import PeftModelfrom transformers import AutoModelForCausalLM base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-4B-Base")model = PeftModel.from_pretrained(base, "while-ai/paper-zero-rl-format-reward-4b")  # the headline armmodel = PeftModel.from_pretrained(base, "while-ai/paper-zero-rl-format-reward-4b", subfolder="baseline")  # another arm

Reproduce

bash

git clone https://github.com/whilehq/whileai-sdk && cd whileai-sdk/recipes/papers/zero-rl-format-rewardpython recipe.py

The recipe README pins the seed, the library versions and the GPU, and its Checks table says what the eval verified. Read the Learned section before quoting a number from this card.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

while-ai

Model Tree

Base

Qwen/Qwen3.5-4B-Base

Adapter

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality