JaeWooShin

qwen2.5-math-1.5b-fewgt-u1-n256-wronglabel-lam1-s0

README

License: apache-2.0

What this is

Qwen2.5-Math-1.5B trained for exactly 300 GRPO steps. Training both starts at and is anchored to the original base weights W0. Two things differ from an ordinary RLVR run.

The reward is spurious. compute_score_wronglabel pays 1.0 only when the parseable \boxed{} answer equals a deterministic wrong target, the ground truth plus one. Non-numeric ground truths have no wrong target, so this signal is far sparser than the inverted one. Either way the reward is computed against the real ground-truth answer, so it consumes labels (see the budget below).

The update is confined to a rank-one band. The loss carries lambda/2 * sum ||(I - u1 u1^T)(W - W0)||_F^2 with lambda = 1, which penalizes any part of the weight change lying outside a single direction u1, in each of the 196 target matrices. That u1 was measured from a separate donor run: 256 real ground-truth labels, true correctness reward, 300 steps. Only the direction crossed over. The donor's weights were never loaded here, and the anchor stays W0.

Observed result

Accuracy on six math benchmarks, sampled at temperature 0.6, top-p 0.95, up to 3072 response tokens, scored by boxed-answer exact match after normalization. AIME and AMC are averaged over 32 samples per problem, the rest over 8.

Table with columns: benchmark, trials, base W0, inverted, wrong-label
benchmark	trials	base W0	inverted	wrong-label
AIME24	32	0.046	0.056	0.054
AIME25	32	0.032	0.047	0.055
AMC23	32	0.265	0.399	0.412
MATH500	8	0.307	0.527	0.551
Minerva	8	0.051

MATH500 during training, greedy, one sample per problem:

Table with columns: step, accuracy
step	accuracy
0	0.304
25	0.33
50	0.386
75	0.47
100	0.524
125	0.514
150	0.572
175	0.582
200	0.594
225

While this was happening the training reward paid out on roughly under 2 percent of samples, and by step 300 the fraction of weight-change energy lying outside the u1 band was 0.0058.

No control arm was run alongside these. This batch contains the two spurious rewards with the band, and the base model. It does not contain the same rewards without the band, nor the band with an ordinary correctness reward. So these numbers show what happened; attributing it to the band rather than to some other part of the setup needs those missing arms.

Training setup

Table with columns: setting, value
setting	value
base / anchor	W0, `Qwen/Qwen2.5-Math-1.5B` at `4a83ca6e4526a4f2da3aa259ec36c259f66b2ab2`
steps	300 (exact; the launcher rejects a short run)
advantage estimator	grpo
reward function	`compute_score_wronglabel`
prompt batch / samples per prompt	64 / 16
learning rate	5e-07
prompt / response tokens	1024 / 3072

Label budget, please read before citing

A 300-step run draws 300 x 64 = 19,200 distinct prompts without replacement under the fixed data seed, and reads each one's ground-truth answer to score the reward. This arm therefore consumes up to 19,200 labels, not the 256 that produced u1. It is a mechanism probe with a labeled reward. It is not a label-free method and not a few-gold method.

Use

python

from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "JaeWooShin/qwen2.5-math-1.5b-fewgt-u1-n256-wronglabel-lam1-s0"tok = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto") prompt = "Solve the following math problem step by step. Put your final answer within \\boxed{}.\n\n"messages = [{"role": "user", "content": prompt + "What is 17 * 23?"}]ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)print(tok.decode(model.generate(ids, max_new_tokens=512)[0][ids.shape[-1]:], skip_special_tokens=True))

The tokenizer's chat template supplies Qwen2.5-Math's own system prompt, which is what evaluation used.

Limitations

Trained by a reward that opposes or ignores correctness. Its behaviour away from short-form boxed math is untested, and nothing here was checked for safety, instruction following or general capability.
Single seed, single model, single label count, single lambda.
Weights are bfloat16, cast from the fp32 training master weights during the merge. verl's own model_merger test confirmed the merged model matches the native checkpoint.

Reproducing

Everything came from a portable transfer bundle:

text

launch.py --phase guided --model qwen --labels 256 --reward wronglabel --gpu 0 --execute

, then evaluate.py for the battery. Configs, logs, curves and the validation generations are in the companion dataset repo.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

JaeWooShin

Model Tree

Base

Qwen/Qwen2.5-Math-1.5B

Fine-tuned

this model

Input Modalities

Text

Output Modalities