ssurface

cot-dialect-qwen3-4b-instruct-grpo-l4

README

License: apache-2.0

Results

Table with columns: Accuracy
	Accuracy
After SFT	73.7%
After GRPO (this adapter)	86.0%
Difference	+12.3 pp

GSM8K test (n=1317), greedy decoding, single-turn, no exemplars, no self-consistency.

Training data

GSM8K train, re-expressed at level L4 by a teacher model: 6976 examples, median chain length 41 characters inside <think>.

Across the family the median chain runs from 532 characters at L1 to 16 at L5 — a 33x span. An L4 chain looks like this:

markdown
K=18*2.5;D=8*4;T=K+D->T=77

Training setup

GRPO on top of the merged level-4 SFT model.

Table

Engine	`trl.GRPOTrainer` on stock `transformers`, attention `sdpa`
Reward	`correctness`, `format`, `gr3`
Loss type	dapo
Generations per prompt	8
Batch	16 x 1 accum
Max completion	256 tokens
Learning rate	1e-05
KL coefficient (beta)

Reward components

correctness — +/- the gold solution's step count on an answer match, so harder problems are worth more
format — the response must be one <think>...</think> block then #### <answer>
gr3 — multiplicative length rescaling of the positive combined reward, floored at 0.3 -- it scales rewards that are already positive, so it cannot reorder correct above incorrect

Engine note. Stock transformers with sdpa attention, not a fused-kernel wrapper. The fused path produced adapters whose lora_B matrices were all zero — mathematically inert despite loading without error. Every adapter in this collection was verified lora_B != 0 before publishing; 13 that failed that check were withheld.

Usage

markdown
Solve this using Level 4 (Shorthand).
Problem: {your problem}

Stacks on the SFT model, not the raw base. Trained against the merged SFT model, so loading it straight onto Qwen/Qwen3-4B-Instruct-2507 will not reproduce the number above.

python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B-Instruct-2507", torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "ssurface/cot-dialect-qwen3-4b-instruct-sft-l4")   # 1. SFT for this level
model = model.merge_and_unload()
model = PeftModel.from_pretrained(model, "ssurface/cot-dialect-qwen3-4b-instruct-grpo-l4")   # 2. this adapter
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-4B-Instruct-2507")

Limitations

Trained and evaluated on math word problems only.
Accuracy falls with problem difficulty, fastest at the compressed levels.
Single seed unless the repo name says otherwise; differences of a couple of points are within noise (95% half-width ~2.7 pp at n=1317, ~4.4 pp at n=500).

Citation

bibtex
@misc{cot-compression-dialects,
  title  = {Chain-of-Thought Compression Dialects},
  author = {Frolov, Anatolii},
  year   = {2026}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

ssurface

Model Tree

Base

Qwen/Qwen3-4B-Instruct-2507

Adapter

this model

Input Modalities

Text

Output Modalities