Kiffaz11

Qwen3-0.6b-ifrlvr-grpo

README

License: apache-2.0

Results

Same-harness comparison against the original Qwen/Qwen3-0.6B. IFBench and IFEval are prompt-level accuracy (strict / loose).

Non-thinking

2048-token budget. GSM8K and MMLU use standard few-shot prompting with no chat template.

Table with columns: Benchmark, base, grpo-C, Δ
Benchmark	base	grpo-C	Δ
IFBench (strict / loose)	16.67 / 22.00	23.33 / 27.67	+6.67 / +5.67
IFEval (strict / loose)	31.05 / 31.98	31.98 / 33.64	+0.93 / +1.66
GSM8K	41.55	46.47	+4.92
MMLU	39.97	40.00	+0.03

Thinking

Chat template (which enables Qwen3's reasoning block), 8192-token budget.

Table with columns: Benchmark, base, grpo-C, Δ
Benchmark	base	grpo-C	Δ
IFBench (strict / loose)	20.33 / 26.00	26.33 / 31.33	+6.00 / +5.33
IFEval (strict / loose)	57.30 / 60.81	64.70 / 69.13	+7.40 / +8.32
GSM8K	62.93	60.42	−2.51 (n.s.)
MMLU (zero-shot CoT)	47.51	45.82

Significance

Paired McNemar exact test on stored per-prompt outputs, with a paired bootstrap confidence interval (20 000 resamples). Benchmarks scored through lm_eval have no per-item log, so they use a conservative unpaired two-proportion test.

Table with columns: Comparison, Δ, 95% CI, p
Comparison	Δ	95% CI	p
IFBench strict · non-thinking	+6.67	[+3.33, +10.33]	3.3e-04
IFBench strict · thinking	+6.00	[+2.00, +10.33]	7.9e-03
IFEval strict · thinking	+7.39	[+3.33, +11.65]	5.7e-04
IFEval loose · thinking	+8.13	[+3.88, +12.38]	1.8e-04

Every instruction-following gain is significant. Nothing regresses significantly.

Caveats, stated plainly

The math gain is a non-thinking gain. GSM8K rises +4.92 without reasoning, but with reasoning enabled both models land together near 60–63. Reasoning already supplies what the 20% GSM8K slice of the RL mix bought.
Both thinking-mode reasoning benchmarks sit nominally below base (GSM8K −2.51, MMLU-CoT −1.69). Neither is statistically significant and both confidence intervals contain zero, so this is not a demonstrated regression — but it is two independent negatives in the same direction and deserves a tighter measurement.
Read thinking-mode GSM8K on flexible-extract. The strict-match filter requires a literal #### n ending that a reasoning response never emits; it scores formatting, not arithmetic (base 15.92 / grpo-C 9.33 under that filter).
Thinking-mode MMLU is the generative zero-shot-CoT variant (mmlu_flan_cot_zeroshot, 57 subjects, 1065 items). The default MMLU task is scored by log-likelihood over the four options — nothing is generated, so "thinking" is undefined for it. Both models use the identical subsample and seed.

Harness calibration against published Qwen3-0.6B figures: our base measurement gives IFBench-loose 26.00 (published 25.67) and IFEval-strict 57.30 (published 59.89).

Training

Table

Init	a task-arithmetic merge of two earlier GRPO runs (WARP-style restart, not from base)
Algorithm	GRPO (TRL), 1200 steps, 8 generations per prompt
Data	`allenai/IF_multi_constraints_upto5` (80%) + GSM8K (20%)
Reward	IFEvalG constraint verifiers; GSM8K exact answer
KL β	0.02 — realised KL mean 0.040, max 0.164
Reward curve	0.551 → 0.631, flat across the final 300 steps
Seed	17
Hardware	1× H100, ~100 minutes

Prompts were reward-density prefiltered and 8-gram decontaminated against the evaluation sets: 7420 probed → 5389 kept (941 saturated, 1090 hopeless, 580 decontamination drops). Prompts the policy already always solves or never solves are dropped, since neither produces gradient under GRPO's group-relative advantage.

Interpolating this checkpoint back toward base at α=0.75 reduced IFBench by 1.0 and GSM8K by 2.0, so the raw RL checkpoint is the one released.

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Kiffaz11/qwen3-0.6b-ifrlvr-grpo-C"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

messages = [{"role": "user", "content": "Write exactly three sentences about the sea."}]
text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
)

Set enable_thinking=True for the reasoning mode measured above.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Kiffaz11

Model Tree

Base

Qwen/Qwen3-0.6B

Fine-tuned

this model

Input Modalities

Text

Output Modalities