Result, from the recipe README
Table with columns: comparison, target delta, verdict| comparison | target delta | verdict |
|---|
| baseline vs base | +0.029 [-0.004, +0.063] | flat (noise < 0.142) |
| method vs base | +0.023 [-0.011, +0.058] | flat, and over-optimized |
| method vs baseline | -0.005 [-0.041, +0.029] | flat, no difference |
The method dropped 77.5% of groups against the baseline's 6.2%, moved covered_all +0.090 and the training reward +0.064, and did not move the target. On the probed rows the target went -0.058 [-0.108, -0.017]. About 73 GPU minutes on H100s.
Arms in this repo
The root holds the arm the recipe README's headline number reports. Every other arm is a subfolder named after it. checkpoints/ never ships.
Table with columns: folder, arm| folder | arm |
|---|
. | method: flat-group test on the binary outcome |
baseline | baseline: flat-group test on the shaped score |
eval | eval rows for base, baseline and method |
Load
from peft import PeftModelfrom transformers import AutoModelForCausalLM base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-1.7B")model = PeftModel.from_pretrained(base, "while-ai/community-airline-voice-outcome-filter-1.7b") # the headline armmodel = PeftModel.from_pretrained(base, "while-ai/community-airline-voice-outcome-filter-1.7b", subfolder="baseline") # another arm
Reproduce
git clone https://github.com/whilehq/whileai-sdk && cd whileai-sdk/recipes/community/airline-voice-concise-under-probe-outcome-filterpython run.py
The recipe README pins the seed, the library versions and the GPU, and its Checks table says what the eval verified. Read the Learned section before quoting a number from this card.