liangzhidanta

Qwen3-8B-CC-SFT-v2

README

License: apache-2.0

Key Result

Best observed canonical303 Pass@1: 52.48% (159/303) — 4090 run of this same checkpoint at T=0.5 (temperature selected on an independent dev set; see the table below).

Best observed configurations of this same checkpoint:

Table with columns: Configuration, Hardware, Solved, Pass@1
Configuration	Hardware	Solved	Pass@1
T=0.5 (dev-set selected)	4090 (TP=2)	159/303	52.48%
T=0.3, independent reproduction	A100-80G (TP=1)	154/303	50.83%
T=0.3, original paired run	4090 (TP=2)	144/303	47.52%

T=0.5 was selected on an independent 60-task development set (task-id and repo disjoint from canonical303), then evaluated once on canonical303 — no post-hoc selection on this benchmark.
The T=0.5 run solved +15 tasks vs the original T=0.3 run (paired McNemar p = 0.18; 303/303 completed, 1 environment failure counted as unsolved) — a positive but not statistically significant difference at single-rollout power.

Same-harness paired comparison on the original 4090 runs — the clean Base → V1 → V2 training-improvement evidence:

Internal canonical evaluation: 303 SWE-smith-derived, repo-disjoint tasks, Claude Code native context compaction ON, Pass@1, 1 rollout per task. Not SWE-bench Verified.

Table with columns: Model, Training Stage, Solved / 303, Pass@1
Model	Training Stage	Solved / 303	Pass@1
Qwen3-8B	Base	10/303	3.30%
Qwen3-8B-CC-SFT-v1	General Agent SFT	106/303	34.98%
Qwen3-8B-CC-SFT-v2	Compact-aware Continued SFT	144/303	47.52%

The same v2 checkpoint achieved 154/303 (50.83%) Pass@1 in an independent A100 reproduction run, compared with 144/303 (47.52%) in the original 4090 run.

V2 vs V1 (paired, identical 303 tasks, same 4090 harness):

+38 solved tasks
+12.54 percentage points
McNemar exact test p = 0.0002
95% paired bootstrap CI = [6.3, 18.8] percentage points

Cross-Machine Reproduction

The same SFT-v2-E1 model weights were re-evaluated on the identical canonical303 task set on independent A100 hardware:

Table with columns: Run, Hardware, Solved, Pass@1
Run	Hardware	Solved	Pass@1
Original reference	4090 (TP=2 serving)	144/303	47.52%
Independent reproduction	A100-80G (TP=1 serving)	154/303	50.83%

Held identical: SFT-v2-E1 checkpoint (weights unchanged), canonical303 task identities, REAL40K context, Claude Code native compaction, temperature 0.3, top_p 0.95, max 32 agent calls, Pass@1, 1 rollout per task. The A100 run completed 303/303 valid environments with 0 infrastructure failures. Observed difference: +10 solved tasks (+3.30 pp); Wilson 95% CI for the A100 run: [45.22%, 56.41%].

The model weights are unchanged — this is a cross-machine performance reproduction, not a bitwise-identical, trajectory-identical, or runtime-identical execution.

Successor: Qwen3-8B-CC-SFT-v2-2e — a second epoch on the same compact-focused recipe — reaches 166/303 (54.79%, T=0.4) on canonical303.

Reproduction Notes

Runtime-level serving differences existed between the two environments:

Table with columns: Metric, 4090 reference, A100 reproduction
Metric	4090 reference	A100 reproduction
Serving parallelism	TP=2	TP=1
Compaction trigger rate	239/303 (78.9%)	81.9%
Compact events per triggered task	2.65	3.84
Tasks with autocompact thrashing	84	121

The compact-fix implementation also differs slightly between the two serving stacks. Runtime-level compaction frequency differed between the two serving environments, so this experiment should be interpreted as a cross-machine performance reproduction rather than trajectory-level deterministic reproduction.

Paired Evaluation

Because both models were evaluated on exactly the same 303 tasks with the same protocol, per-task outcomes can be paired:

Table with columns: Paired outcome, Tasks
Paired outcome	Tasks
Both solved	76
V2 only	68
V1 only	30
Neither	129

Net +38 tasks in favor of v2 (McNemar discordant pairs 68 vs 30; exact p = 1.56e-4, reported as 0.0002).

Why v2?

Trajectory-level failure analysis of v1 under Claude Code native context compaction showed that, after a compaction event, the agent frequently lost or corrupted execution state, including:

completed vs pending work (what was done vs still to do)
modification state (which files / symbols were edited)
test execution state and test results
the current hypothesis and the next action

v2 specifically targets agent-state compression and post-compaction continuation. It is not "more data + another SFT": every added supervision segment is derived from a real native-compaction boundary.

Training Data

Standalone dataset: liangzhidanta/claude-code-glm53-swesmith-compact-trajectories

Two core supervision types:

Compact Summary — the real Claude Code native compact request is forwarded to the teacher (GLM-5.3), and the teacher-generated state summary becomes the training target. Goal: faithful state compression, especially done / not-done status, modification state, test state, current hypothesis, and pending action.
Post-Compaction Continuation — the real model-visible post-compaction context is the prefix, and the subsequent assistant reasoning / text / tool calls are the target. Goal: learn to continue execution from a compressed state.

Dataset statistics (from the published dataset manifest)

Table with columns: Level, Count
Level	Count
Source trajectories	189 (138 one-compact, 51 two-compact)
Native compact events	240
Context-correct segments	890 = 240 compact-summary + 403 post-compaction + 247 optional pre-compact
Compact-focused segments	643 = 240 + 403
Deployment-aligned ≤40K subset	626 (239 + 387; the remaining 17 segments are long-context auxiliary)

Training recipe used for this checkpoint (COMPACT_FOCUSED, ≤40960 tokens)

Table with columns: Component, Count, Description
Component	Count	Description
V1 replay	1003	Original execution-verified v1 agent trajectories
Compact-summary segments	239	Real CC compact request → GLM-5.3 summary
Post-compaction continuation segments	387	Real post-compaction visible context → agent continuation
Total	1629	All segments ≤40960 tokens

Context-Correct Training Format

A naive conversion concatenates pre-compaction history + summary + post-compaction history into one training context. At real inference time, however, the pre-compaction trimmed history is no longer visible to the model — so the naive full-concat view creates a train–inference context mismatch and leaks history that the deployed model will never see.

SFT-v2 therefore constructs context-correct segments: each training example is split at real compaction boundaries, and each prefix is rebuilt from the actual model-visible wire request at that point. Post-compaction targets are trained only against inference-visible state.

Native Compaction Behavior

Measured over the canonical303 evaluation (Claude Code native auto-compaction ON):

Table with columns: Metric, V1, V2
Metric	V1	V2
Tasks triggering compaction	238/303 (78.5%)	239/303 (78.9%)
Compact events	752	633
Events per compact-triggered task	3.16	2.65
Tasks with autocompact thrashing	83	84
Pass@1	34.98%	47.52%

Trigger rate is essentially unchanged: v2 does not improve by avoiding compaction. Instead, under nearly identical compaction-trigger rates, v2 requires fewer compaction events per triggered task (3.16 → 2.65, ≈16.2% relative reduction) while achieving substantially higher task success — consistent with more efficient post-compaction continuation. Thrashing (83 → 84 tasks) is not improved and remains an open limitation. For reference, the Qwen3-8B base model triggered compaction on 124/303 tasks (227 events).

Training Details

Table with columns: Value
	Value
Initial checkpoint	Qwen3-8B-CC-SFT-v1 (continued SFT, initialized from v1)
Framework	slime v0.3.2 (Megatron-LM backend)
Fine-tuning type	Full-parameter continued SFT
Dataset	COMPACT_FOCUSED recipe, 1629 segments (see above)
Epochs	1
Optimizer	Adam (β₁=0.9, β₂=0.95), weight decay 0.1
Learning rate	5e-6, cosine decay to 5e-7
Warmup	0.1
Context	40960 (deployment REAL40K; silent truncation disabled)
Steps	367 optimizer steps
Batch size	rollout batch 4 / global batch 4
Hardware	8× RTX 4090 24GB
Parallelism	TP=8, DP=1, PP=1, CP=1, sequence parallel

Losses are reported as training information only, not as capability metrics.

Training curve

Masked SFT loss over 367 optimizer steps (1 epoch):

Evaluation Protocol

Table with columns: Item, Value
Item	Value
Task set	canonical303 (303 tasks)
Task source	SWE-smith-derived, repo-disjoint from training repositories
Environment	303/303 valid Docker environments
Context	40960 tokens (CC-visible context = SGLang real context)
Context management	Claude Code native automatic context compaction (ON)
Sampling	temperature = 0.3, top_p = 0.95
Max agent calls	32
Metric	Pass@1, 1 rollout per task
Harness	Claude Code 2.1.258 compatible harness
Serving	SGLang 0.5.15.post1 (inference TP=2; independent of training parallelism)

No-compaction runs are treated as diagnostic controls, not the primary deployment-aligned evaluation: the target agent environment uses Claude Code native context compaction. SWE-bench Verified results are not reported in this release; the results above use our SWE-smith-derived canonical evaluation.

Model Lineage

text

Qwen/Qwen3-8B        ↓execution-verified GLM-5.3 trajectories        ↓Qwen3-8B-CC-SFT-v1        ↓trajectory-level failure analysis (native compaction failure modes)        ↓compact-aware targeted supervision (189 sources → 240 native compact events)        ↓context-correct continued SFT        ↓Qwen3-8B-CC-SFT-v2        ↓verifiable GRPO (planned)

Usage

python

from transformers import AutoTokenizer, AutoModelForCausalLM model_id = "<this-repo>"tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto")

This checkpoint uses the original Qwen3 tokenizer and chat template. For coding-agent evaluation, the model was served through SGLang with a Qwen3 reasoning parser and tool-call parser, and used behind a Claude-Code-compatible harness with native context compaction enabled.

Limitations

Task pool shared with v1 (same tasks, different-trajectory supervision).
Compact summaries are generated by GLM-5.3 and may contain residual errors.
84 of 303 tasks still exhibit autocompact thrashing (unchanged vs v1's 83); thrashing is not solved by v2.
Evaluated only at temperature 0.3.
The internal 303-task benchmark is not SWE-bench Verified; SWE-bench Verified results are not reported in this release.
Performance may depend on harness/tool configuration; tool-use performance should not be interpreted as standalone raw-model benchmark performance.

Citation

bibtex

@model{qwen3_8b_cc_sft_v2,  title={Qwen3-8B-CC-SFT-v2: Compact-Aware Coding Agent},  author={liangzhidanta},  year={2026},  url={https://huggingface.co/liangzhidanta/Qwen3-8B-CC-SFT-v2}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider