code-critic-model

Qwen3-4B-Critic-SFT

README

License: apache-2.0

Where it appears in the paper

Table with columns: Paper location, Row label
Paper location	Row label
Table 1, every agent block	`Qwen3-4B + SFT`
Table 3, corpus ablation	`4B, Qwen+CWM`
Section 2.4 and Figure 3	the SFT checkpoint that DPO starts from

Original run name: qwen3-4b-instruct-2507-full-sft-prm-r2egym-swebench-instructions-k5-cwm-plus-qwen-opus-distill-32k-multiturn. The repository code-critic-model/qwen3-4b-sft-prm holds a byte-identical copy of these weights under the name the DPO runs referenced.

Training data

code-critic-model/critic-sft-cwm-qwen, 6,447 examples.

Tasks: 500 R2E-Gym instances from matplotlib, moto, and sympy, disjoint from SWE-bench Verified.
Agents that produced the trajectories: CWM-32B (500 trajectories, 4,532 examples) and Qwen3-Next-80B-A3B-Instruct (483 trajectories, 1,915 examples).
Teacher: Claude Opus 4.6, queried every 5 agent steps with the high-level prompt.

Training setup

Full-parameter SFT with LLaMA-Factory. The config is finetuning/qwen3_4b_critic_full_sft_l40s_train_multiturn_resumable.yaml in the repository.

Table with columns: Setting, Value
Setting	Value
Base model	Qwen/Qwen3-4B-Instruct-2507
Chat template	`qwen3_nothink`
Sequence length	32,768 tokens
Loss	final critique turn only (`mask_history: true`)
Hardware	8 x L40S, per-device batch 1, effective batch 8
Optimizer	AdamW, lr 5e-6, cosine schedule, warmup ratio 0.1
Epochs	3
Precision

Results

Resolve rate on SWE-bench Verified (500 instances), from Table 1 of the paper, best of k=5 and k=10 per configuration. The untrained base model is included so the effect of SFT is visible.

Table with columns: Coding agent, No critic, Untrained Qwen3-4B-Instruct-2507, + Qwen3-4B-Critic-SFT, + Qwen3-4B-Critic-SFT-DPO
Coding agent	No critic	Untrained Qwen3-4B-Instruct-2507	+ Qwen3-4B-Critic-SFT	+ Qwen3-4B-Critic-SFT-DPO
Qwen3-32B	8.8	10.2	11.4	14.4
Qwen3-Next-80B-A3B	20.0	20.2	24.2	26.2
GPT-OSS-20B	3.0	6.8	9.8	14.8

How to use

Serve with vLLM in bf16 and run an agent through the repository's mini-swe-agent fork, which inserts a critique every k steps.

bash
vllm serve code-critic-model/Qwen3-4B-Critic-SFT \
    --served-model-name Qwen3-4B-Critic-SFT \
    --dtype bfloat16 --max-model-len 65536 --port 8071

bash scripts/run_critic_max150.sh prm_issue_res_instructions_step_aware 5 0 qwen3-80b \
    --prm Qwen3-4B-Critic-SFT --prm-node <vllm-host>:8071 --slice :500 \
    --prefix-dir <path to the matching no-critic run>

The --prm name goes to LiteLLM, which needs a matching entry in mini-swe-agent/configs/litellm_model_registry.json to price the calls. Copy the block for the original run name to a new key Qwen3-4B-Critic-SFT, or serve under the original run name. Without an entry the critic call fails and the agent runs without critiques.

To call the critic directly, take any training record, drop its final teacher critique, and generate. The system message and trajectory encoding in the records are exactly what the model saw in training. A complete snippet is on the Qwen3-8B-Critic-SFT card; only the repo name changes.

Citation

bibtex
@misc{gandhi2026steerdontsolvetraining,
  title={Steer, Don't Solve: Training Small Critic Models for Large Code Agents},
  author={Shubham Gandhi and Yiqing Xie and Atharva Naik and Ruichen Zhu and Carolyn Rose},
  year={2026},
  eprint={2606.21811},
  archivePrefix={arXiv},
  primaryClass={cs.SE},
  url={https://arxiv.org/abs/2606.21811}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

code-critic-model

Model Tree

Base

Qwen/Qwen3-4B-Instruct-2507

Fine-tuned

this model

Input Modalities

Text

Output Modalities