Qwen3-4B-GRPO-119 API & Inference Endpoint

📈 Training Curves

Top-left: AIME 2024 pass@1 every 20 iterations (gray band = ±1pp test-retest noise, calibrated by 3 baseline runs). Top-right: train-batch reward rises while mean response length falls — accuracy and efficiency improve together. Bottom-left: truncation ratio halves on both train (8k cap) and eval (16k cap). Bottom-right: rollout dominates ~85–96% of each ~23-min iteration.

🏆 Key Results (120 GRPO iterations)

Table with columns: Metric, Base (Qwen3-4B), This model (iter 119), Δ
Metric	Base (Qwen3-4B)	This model (iter 119)	Δ
AIME 2024 pass@1 (30 problems × 16 samples)	57.5%	65.0% (peak 66.0% @ iter 79)	+7.5pp
Train batch reward (DAPO-Math-17K)	0.535	0.629	+0.094
Response length median	7,400 tok	3,777 tok	−49%
Response length mean	6,403 tok	4,476 tok	−30%
Truncated ratio (train, 8k cap)	46.1%	21.1%	−54%
Truncated ratio (eval, 16k cap)	37.5%	19.6%	−48%
Degenerate repetition	0.0%	0.0%	—

The headline finding: the model became more accurate and more concise simultaneously — no length-based reward hacking, entropy stays healthy (0.34 → 0.29).

Table with columns: Iteration, 0 (base), 19, 39, 59, 79, 99, 119
Iteration	0 (base)	19	39	59	79	99	119
pass@1	57.5%	60.6%	61.0%	63.1%	66.0%	64.8%	65.0%

Test-retest noise: ±1pp (baseline measured 3×: 57.5 / 57.9 / 58.5%)
Fast gains in the first 80 iterations (+8.5pp), then a 65%±1pp plateau — training stopped at 120 based on this

⚙️ Training Setup

Table

Algorithm	GRPO — group-relative advantage, group size G=8 (32 prompts × 8 samples = 256 trajectories/iteration = 1 optimizer step)
Clipping	Asymmetric DAPO-style: eps-clip 0.2 / eps-clip-high 0.28; no KL penalty (coef 0), no entropy bonus
Reward	Rule-based verifier (deepscaler RM): extract `\boxed{}` after `</think>`, grade vs ground truth via mathd/sympy symbolic equivalence → 0/1
Data	zhuzilin/dapo-math-17k, 32 prompts/iter ≈ 5.5 epochs over the run
Optimizer	Adam, lr 1e-6 constant, wd 0.1, β=(0.9, 0.98)

The official recipe targets 8×H100 80GB. Adapting it to 8×RTX 4090 24GB required (full story in EXPERIMENT_REPORT.md):

TP 2→4: a single truncated sample (~8.4k tokens) materializes a ~5GiB fp32 full-vocab logits block in the log-prob path — halving the vocab shard per GPU was the decisive fix;
Token budget 9216→4096 + log-probs-chunk 1024: logits buffers scale linearly with microbatch tokens; chunking bounds softmax temporaries;
Disabled the (coefficient-zero) KL reference forward — pure waste of memory and time;
expandable_segments:True against fragmentation (requires the +slime fork of torch_memory_saver);
Session/cluster survival (loginctl enable-linger + tmux) and checkpoint retention (53GB each, keep-latest) for a multi-day run.

🚀 Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "liangzhidanta/Qwen3-4B-GRPO-119",
    torch_dtype=torch.bfloat16, device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("liangzhidanta/Qwen3-4B-GRPO-119")

messages = [{"role": "user",
             "content": "Find the number of ordered pairs of prime numbers $(p, q)$ "
                        "such that $3p + 5q = 102$. Put the answer in \\boxed{}."}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)

out = model.generate(**inputs, max_new_tokens=8192,
                     do_sample=True, temperature=1.0)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Best served with thinking mode enabled (default chat template) and a generous generation budget; the model was trained at temperature 1.0.

⚠️ Limitations

Trained and evaluated on English math reasoning only (DAPO-Math-17K / AIME 2024);
21% of training rollouts still hit the 8192-token cap;
Checkpoint at iteration 119 (plateau representative); the iter-79 peak (66.0%) is within the ±1pp noise band and was not retained;
~2/3 of prompt groups produced zero-variance rewards each iteration (a DAPO dynamic-sampling opportunity — see report §8).

中文版本

GRPO 数学推理强化学习 —— 基于 slime 框架、在 8×RTX 4090 上端到端训练

AIME 2024：57.5% → 65.0%（+7.5 个百分点）· 响应长度 −49% · 截断率 −54%

📈 训练曲线

左上：每 20 轮一次的 AIME 2024 pass@1（灰色带 = ±1pp 复测噪声，由 3 次基线重复测量标定）。右上：训练批奖励上升的同时平均响应长度下降——正确率与效率同步改善。左下：训练（8k 上限）与评估（16k 上限）的截断率均减半。右下：每轮约 23 分钟中 rollout 占 85–96%。

🏆 核心结果（120 轮 GRPO 训练）

Table with columns: 指标, 基座（Qwen3-4B）, 本模型（第 119 轮）, 变化
指标	基座（Qwen3-4B）	本模型（第 119 轮）	变化
AIME 2024 pass@1（30 题 × 16 采样）	57.5%	65.0%（峰值 66.0% @ 第 79 轮）	+7.5pp
训练批平均奖励（DAPO-Math-17K）	0.535	0.629	+0.094
响应长度中位数	7,400 tok	3,777 tok	−49%
响应长度均值	6,403 tok	4,476 tok

核心发现：模型变得更准的同时变得更简洁——没有出现"靠写长碰答案"的 reward hacking，策略熵保持健康（0.34 → 0.29）。

Table with columns: 轮次, 0（基线）, 19, 39, 59, 79, 99, 119
轮次	0（基线）	19	39	59	79	99	119
pass@1	57.5%	60.6%	61.0%	63.1%	66.0%	64.8%	65.0%

复测噪声：±1pp（基线测了 3 遍：57.5 / 57.9 / 58.5%）
前 80 轮贡献几乎全部增益（+8.5pp），之后进入 65%±1pp 平台期——据此在第 120 轮停止训练

⚙️ 训练配置

Table

算法	GRPO——组内相对 advantage，组大小 G=8（32 prompt × 8 采样 = 256 条轨迹/轮 = 1 个优化器步）
裁剪	DAPO 式非对称裁剪：eps-clip 0.2 / eps-clip-high 0.28；无 KL 惩罚（系数 0）、无熵加成
奖励	规则判分器（deepscaler RM）：取 `</think>` 后正文的 `\boxed{}`，与标准答案做 mathd/sympy 符号等价比对 → 0/1
数据	zhuzilin/dapo-math-17k，每轮 32 个 prompt，全程约 5.5 个 epoch
优化器	Adam，lr 1e-6 恒定，wd 0.1，β=(0.9, 0.98)
精度	bf16 混合精度（梯度 fp32 all-reduce，优化器/主权重 fp32）

官方配方按 8×H100 80G 设计，适配 8×RTX 4090 24G 做了以下改造（完整记录见 EXPERIMENT_REPORT.md）：

TP 2→4：单条截断样本（约 8.4k token）在 log-prob 路径会物化约 5GiB 的 fp32 全词表 logits——把词表分片再对半是决定性修复；
token 预算 9216→4096 + log-probs-chunk 1024：logits 缓冲与 microbatch token 数成正比，分块限制 softmax 临时缓冲；
关闭（系数为零的）KL 参考前向——纯粹的显存与时间浪费；
expandable_segments:True 对抗碎片（需要 torch_memory_saver 的 +slime fork）；
多日训练的会话/集群存活（loginctl enable-linger + tmux）与 checkpoint 保留策略（每份 53GB，只留最新）。

🚀 使用方法

python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "liangzhidanta/Qwen3-4B-GRPO-119",
    torch_dtype=torch.bfloat16, device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("liangzhidanta/Qwen3-4B-GRPO-119")

messages = [{"role": "user",
             "content": "求满足 3p + 5q = 102 的质数有序对 (p, q) 的个数。答案写在 \\boxed{} 中。"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)

out = model.generate(**inputs, max_new_tokens=8192,
                     do_sample=True, temperature=1.0)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

建议开启思考模式（默认 chat template）并给足生成长度；模型训练温度为 1.0。

⚠️ 局限性

仅在英文数学推理数据上训练和评估（DAPO-Math-17K / AIME 2024）；
21% 的训练 rollout 仍会触及 8192 token 上限；
采用第 119 轮 checkpoint（平台期代表）；第 79 轮峰值（66.0%）在 ±1pp 噪声带内、未保留；
每轮约 2/3 的 prompt 组奖励零方差（DAPO dynamic sampling 的改进方向——见报告 §8）。

🙏 致谢

slime——RL 训练框架；
Qwen 团队的优秀基座模型；
DAPO 与 DeepScaler 论文提供的数据、奖励设计与非对称裁剪配方。

📁 仓库文件

Table with columns: 文件, 说明
文件	说明
`model-*.safetensors`	微调权重（第 119 轮，GRPO）
`curves.png`	四联训练曲线图
`EXPERIMENT_REPORT.md`	完整实验报告：配置、指标、工程日志

📈 Training Curves

🏆 Key Results (120 GRPO iterations)

Table with columns: Metric, Base (Qwen3-4B), This model (iter 119), Δ
Metric	Base (Qwen3-4B)	This model (iter 119)	Δ
AIME 2024 pass@1 (30 problems × 16 samples)	57.5%	65.0% (peak 66.0% @ iter 79)	+7.5pp
Train batch reward (DAPO-Math-17K)	0.535	0.629	+0.094
Response length median	7,400 tok	3,777 tok	−49%
Response length mean	6,403 tok	4,476 tok	−30%
Truncated ratio (train, 8k cap)	46.1%	21.1%	−54%
Truncated ratio (eval, 16k cap)	37.5%	19.6%	−48%
Degenerate repetition	0.0%	0.0%	—

The headline finding: the model became more accurate and more concise simultaneously — no length-based reward hacking, entropy stays healthy (0.34 → 0.29).

Table with columns: Iteration, 0 (base), 19, 39, 59, 79, 99, 119
Iteration	0 (base)	19	39	59	79	99	119
pass@1	57.5%	60.6%	61.0%	63.1%	66.0%	64.8%	65.0%

Test-retest noise: ±1pp (baseline measured 3×: 57.5 / 57.9 / 58.5%)
Fast gains in the first 80 iterations (+8.5pp), then a 65%±1pp plateau — training stopped at 120 based on this

⚙️ Training Setup

Table

Algorithm	GRPO — group-relative advantage, group size G=8 (32 prompts × 8 samples = 256 trajectories/iteration = 1 optimizer step)
Clipping	Asymmetric DAPO-style: eps-clip 0.2 / eps-clip-high 0.28; no KL penalty (coef 0), no entropy bonus
Reward	Rule-based verifier (deepscaler RM): extract `\boxed{}` after `</think>`, grade vs ground truth via mathd/sympy symbolic equivalence → 0/1
Data	zhuzilin/dapo-math-17k, 32 prompts/iter ≈ 5.5 epochs over the run
Optimizer	Adam, lr 1e-6 constant, wd 0.1, β=(0.9, 0.98)

The official recipe targets 8×H100 80GB. Adapting it to 8×RTX 4090 24GB required (full story in EXPERIMENT_REPORT.md):

TP 2→4: a single truncated sample (~8.4k tokens) materializes a ~5GiB fp32 full-vocab logits block in the log-prob path — halving the vocab shard per GPU was the decisive fix;
Token budget 9216→4096 + log-probs-chunk 1024: logits buffers scale linearly with microbatch tokens; chunking bounds softmax temporaries;
Disabled the (coefficient-zero) KL reference forward — pure waste of memory and time;
expandable_segments:True against fragmentation (requires the +slime fork of torch_memory_saver);
Session/cluster survival (loginctl enable-linger + tmux) and checkpoint retention (53GB each, keep-latest) for a multi-day run.

🚀 Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "liangzhidanta/Qwen3-4B-GRPO-119",
    torch_dtype=torch.bfloat16, device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("liangzhidanta/Qwen3-4B-GRPO-119")

messages = [{"role": "user",
             "content": "Find the number of ordered pairs of prime numbers $(p, q)$ "
                        "such that $3p + 5q = 102$. Put the answer in \\boxed{}."}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)

out = model.generate(**inputs, max_new_tokens=8192,
                     do_sample=True, temperature=1.0)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Best served with thinking mode enabled (default chat template) and a generous generation budget; the model was trained at temperature 1.0.

⚠️ Limitations

Trained and evaluated on English math reasoning only (DAPO-Math-17K / AIME 2024);
21% of training rollouts still hit the 8192-token cap;
Checkpoint at iteration 119 (plateau representative); the iter-79 peak (66.0%) is within the ±1pp noise band and was not retained;
~2/3 of prompt groups produced zero-variance rewards each iteration (a DAPO dynamic-sampling opportunity — see report §8).

中文版本

GRPO 数学推理强化学习 —— 基于 slime 框架、在 8×RTX 4090 上端到端训练

AIME 2024：57.5% → 65.0%（+7.5 个百分点）· 响应长度 −49% · 截断率 −54%

📈 训练曲线

🏆 核心结果（120 轮 GRPO 训练）

Table with columns: 指标, 基座（Qwen3-4B）, 本模型（第 119 轮）, 变化
指标	基座（Qwen3-4B）	本模型（第 119 轮）	变化
AIME 2024 pass@1（30 题 × 16 采样）	57.5%	65.0%（峰值 66.0% @ 第 79 轮）	+7.5pp
训练批平均奖励（DAPO-Math-17K）	0.535	0.629	+0.094
响应长度中位数	7,400 tok	3,777 tok	−49%
响应长度均值	6,403 tok	4,476 tok

核心发现：模型变得更准的同时变得更简洁——没有出现"靠写长碰答案"的 reward hacking，策略熵保持健康（0.34 → 0.29）。

Table with columns: 轮次, 0（基线）, 19, 39, 59, 79, 99, 119
轮次	0（基线）	19	39	59	79	99	119
pass@1	57.5%	60.6%	61.0%	63.1%	66.0%	64.8%	65.0%

复测噪声：±1pp（基线测了 3 遍：57.5 / 57.9 / 58.5%）
前 80 轮贡献几乎全部增益（+8.5pp），之后进入 65%±1pp 平台期——据此在第 120 轮停止训练

⚙️ 训练配置

Table

算法	GRPO——组内相对 advantage，组大小 G=8（32 prompt × 8 采样 = 256 条轨迹/轮 = 1 个优化器步）
裁剪	DAPO 式非对称裁剪：eps-clip 0.2 / eps-clip-high 0.28；无 KL 惩罚（系数 0）、无熵加成
奖励	规则判分器（deepscaler RM）：取 `</think>` 后正文的 `\boxed{}`，与标准答案做 mathd/sympy 符号等价比对 → 0/1
数据	zhuzilin/dapo-math-17k，每轮 32 个 prompt，全程约 5.5 个 epoch
优化器	Adam，lr 1e-6 恒定，wd 0.1，β=(0.9, 0.98)
精度	bf16 混合精度（梯度 fp32 all-reduce，优化器/主权重 fp32）

官方配方按 8×H100 80G 设计，适配 8×RTX 4090 24G 做了以下改造（完整记录见 EXPERIMENT_REPORT.md）：

TP 2→4：单条截断样本（约 8.4k token）在 log-prob 路径会物化约 5GiB 的 fp32 全词表 logits——把词表分片再对半是决定性修复；
token 预算 9216→4096 + log-probs-chunk 1024：logits 缓冲与 microbatch token 数成正比，分块限制 softmax 临时缓冲；
关闭（系数为零的）KL 参考前向——纯粹的显存与时间浪费；
expandable_segments:True 对抗碎片（需要 torch_memory_saver 的 +slime fork）；
多日训练的会话/集群存活（loginctl enable-linger + tmux）与 checkpoint 保留策略（每份 53GB，只留最新）。

🚀 使用方法

python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "liangzhidanta/Qwen3-4B-GRPO-119",
    torch_dtype=torch.bfloat16, device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("liangzhidanta/Qwen3-4B-GRPO-119")

messages = [{"role": "user",
             "content": "求满足 3p + 5q = 102 的质数有序对 (p, q) 的个数。答案写在 \\boxed{} 中。"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)

out = model.generate(**inputs, max_new_tokens=8192,
                     do_sample=True, temperature=1.0)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

建议开启思考模式（默认 chat template）并给足生成长度；模型训练温度为 1.0。

⚠️ 局限性

仅在英文数学推理数据上训练和评估（DAPO-Math-17K / AIME 2024）；
21% 的训练 rollout 仍会触及 8192 token 上限；
采用第 119 轮 checkpoint（平台期代表）；第 79 轮峰值（66.0%）在 ±1pp 噪声带内、未保留；
每轮约 2/3 的 prompt 组奖励零方差（DAPO dynamic sampling 的改进方向——见报告 §8）。

🙏 致谢

slime——RL 训练框架；
Qwen 团队的优秀基座模型；
DAPO 与 DeepScaler 论文提供的数据、奖励设计与非对称裁剪配方。

📁 仓库文件

Table with columns: 文件, 说明
文件	说明
`model-*.safetensors`	微调权重（第 119 轮，GRPO）
`curves.png`	四联训练曲线图
`EXPERIMENT_REPORT.md`	完整实验报告：配置、指标、工程日志

Qwen3-4B-GRPO-119

README

📈 Training Curves

🏆 Key Results (120 GRPO iterations)

⚙️ Training Setup

🚀 Usage

⚠️ Limitations

中文版本

📈 训练曲线

🏆 核心结果（120 轮 GRPO 训练）

⚙️ 训练配置

🚀 使用方法

⚠️ 局限性

🙏 致谢

📁 仓库文件

Explore FriendliAI today

README

📈 Training Curves

🏆 Key Results (120 GRPO iterations)

⚙️ Training Setup

🚀 Usage

⚠️ Limitations

中文版本

📈 训练曲线

🏆 核心结果（120 轮 GRPO 训练）

⚙️ 训练配置

🚀 使用方法

⚠️ 局限性

🙏 致谢

📁 仓库文件