andresnowak

qwen38-27b-math-grpo-monitor-drgrpo-step100

README

License: apache-2.0

Training

Table

Base model	`Qwen/Qwen3.8-27B`
Algorithm	GRPO with the Dr.GRPO correction (advantages not divided by group std)
Framework	NeMo RL
Data	AceReason-Math
Steps	100
Learning rate	1e-6, cosine decay to 1e-7, 10 warmup iters
Weight decay	0.0
Parallelism	TP=4, PP=2

Reward

text

reward = correctness x (1 + 0.1 * monitor_score)

monitor_score comes from a weak monitor model (HuggingFaceTB/SmolLM3-3B) reading the reasoning trace, weighted at 0.1.

The correctness factor is a floor, not the objective: it stops the model trading away math ability in pursuit of the style term. What the monitor term is meant to change is how the model talks - steering the reasoning toward what the weak monitor finds likely - while accuracy holds. The control arm uses reward = correctness alone and puts no pressure on style.

Token budget

Batch shape is 16 prompts x 8 generations = 128 samples per step, over 100 steps.

Table with columns: per sample, per step, total (100 steps)
	per sample	per step	total (100 steps)
Total tokens (prompt + generated)	4164	~533K	~53.3M
Generated tokens only	4011	~513K	~51.3M

Means are averaged over all 100 steps; prompts account for the ~153-token difference between the two rows.

Limitations

The style half of the objective is not yet verified. Accuracy did hold: reward tracks the control arm closely across all 100 steps, so the correctness floor did its job. The monitor term was provably active (max reward 1.079 vs exactly 1.000 for the control). But whether the reasoning actually moved toward what SmolLM3 finds likely has not been measured on held-out prompts, only observed anecdotally. Do not cite this checkpoint as evidence the method works.
Reasoning length rose slightly relative to the control over training (~4088 vs ~3838 mean generated tokens in the last 10 steps) rather than collapsing, so the style pressure did not compress the chain of thought.
Single seed, single dataset, 100 steps. No safety, refusal, or capability evaluation has been performed.
Inherits all limitations and licensing of the base model.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizerimport torch m = "andresnowak/qwen38-27b-math-grpo-monitor-drgrpo-step100"tok = AutoTokenizer.from_pretrained(m)model = AutoModelForCausalLM.from_pretrained(m, dtype=torch.bfloat16, device_map="auto")

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

andresnowak

Model Tree

Base

Qwen/Qwen3.8-27B

Fine-tuned

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality