Training Pipeline
Qwen/Qwen3-4B-Instruct-2507
→ SFT LoRA (ssurface/qwen3-4b-cot-compress-l4)
→ Merged
→ GRPO with DAPO loss
→ Merged (this model)
Reward Functions
Two reward signals were used during GRPO:
1. Correctness Reward
Checks if the model's final answer (after ####) matches the ground truth.
Reward is scaled by problem complexity (1–5 steps):
Table with columns: Complexity, Reward| Complexity | Reward |
|---|
| 5 (hardest) | 5.0 |
| 4 | 4.0 |
| 3 | 3.0 |
| 2 | 2.0 |
| 1 (easiest) | 1.0 |
| Wrong answer | 0.0 |
2. Goldilocks Length Penalty (double-hinge)
Penalizes <think> blocks that are too short OR too long for the target level.
Table with columns: Level, Style, Target range (chars), Too-short penalty, Too-long penalty| Level | Style | Target range (chars) | Too-short penalty | Too-long penalty |
|---|
| 1 | Verbose | 120–240 | −0.05/char | −0.01/char |
| 2 | Concise | 50–110 | −0.05/char | −0.01/char |
| 3 | Symbolic | 40–100 | −0.05/char | −0.01/char |
This model targets Level 4 (Shorthand): 20-60 chars of reasoning.
GRPO Hyperparameters
Table with columns: Parameter, Value| Parameter | Value |
|---|
| Loss type | DAPO |
| LoRA r | 16 |
| LoRA alpha | 32 |
| Num generations | 8 |
| Max completion length | 256 |
| Batch size | 16 |
| Gradient accumulation | 2 |
| Learning rate | 1e-5 |
| Epochs | 1 |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("ssurface/qwen3-4b-grpo-l4")
tokenizer = AutoTokenizer.from_pretrained("ssurface/qwen3-4b-grpo-l4")
messages = [{"role": "user", "content": "Solve this using Level 4 (Shorthand).\nProblem: ..."}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))