pvdhihihi

Qwen2.5-Math-7B-PSFT-numina-lr5e-5-clip0.2-0.28

README

License: apache-2.0

This checkpoint improves on its base model

Cross-entropy on the held-out validation split (100 Numina test examples, identical at every evaluation): 0.5192 -> 0.4489, -13.5%, best 0.4480 at step 250. This is the lowest validation CE of any run in this series.

Table with columns: step, val CE (original_loss)
step	val CE (`original_loss`)
0 (base)	0.5192
10	0.4859
20	0.4719
50	0.4567
100	0.4516
200	0.4490
300	0.4487
390 (final)	0.4489

PSFT vs SPFT on this model

Same base model, same seed, same world_size (8) and therefore the same DistributedSampler per-rank data ordering, and the same chat-template prompt format. Step 0 reproduced bit-identically (0.5192).

Table with columns: step, PSFT (lr 5e-5), SPFT (lambda 0.2, lr 7e-5)
step	PSFT (lr 5e-5)	SPFT (lambda 0.2, lr 7e-5)
0	0.5192	0.5192
10	0.4859	0.5541
20	0.4719	1.1226
100	0.4516	1.4177
390	0.4489	1.2878

PSFT improves all four base models; SPFT degrades all four

Table with columns: base model, baseline CE, SPFT (lambda 0.2), PSFT
base model	baseline CE	SPFT (lambda 0.2)	PSFT
`meta-llama/Llama-3.2-3B`	0.8149	+86.4%	-20.4%
`meta-llama/Llama-3.1-8B`	0.7367	+127.8%	-23.3%
`deepseek-ai/deepseek-math-7b-base`	0.6727	+95.6%

PSFT's relative gain shrinks as the base model's starting CE improves (-23.3% at 0.737 down to -13.5% at 0.519), consistent with less headroom on a model that already fits this data well.

Mechanism

SPFT weights each token by p * sigmoid(-lambda * delta_logit). As the policy pulls ahead of the frozen reference the sigmoid shrinks, so the objective progressively stops penalising the tokens it should still learn from. Across seven SPFT runs the mean weight began at 0.35-0.41 and collapsed to ~0.15 at every learning rate tried, while unweighted CE roughly doubled. The weighted loss falls monotonically throughout, so it looks like progress.
PSFT uses -min(ratio, clamp(ratio, 1-clip_low, 1+clip_high)), a hard PPO-style clip. Once a token's ratio exceeds 1+clip_high the objective stops rewarding further increase, so drift is bounded without the gradient signal decaying toward zero. Here w_ratio stays at ~0.99999 throughout.

A preceding learning-rate sweep on Llama-3.2-3B (1e-4 / 7e-5 / 5e-5 at lambda 0.2) was monotonic but shallow: doubling the LR moved the outcome only from +86.4% to +107.2%, and all three landed near twice the baseline. That ruled the learning rate out as the dominant cause and pointed at the SPFT weight collapse.

Caveats

weight_mean is not comparable between the two objectives. The trainer logs it as p * sigmoid(...) for SPFT (the actual loss weight, which collapses) but as the raw unclipped ratio exp(log_probs - ref_log_probs) for PSFT, which is a diagnostic that never enters the gradient.
PSFT used weight decay 0.1 (upstream's PSFT default) against the SPFT runs' 0.01, and this pair also differs in learning rate (5e-5 vs 7e-5). The Llama-3.2-3B pair is the clean single-variable comparison (both lr 5e-5). The 3B LR sweep shows 5e-5 -> 7e-5 is worth only ~4 points, far too little to explain a 161-point gap.
train/loss is negative (final -1.029) because PSFT's objective is -min(ratio, clipped) rather than a weighted cross-entropy. Compare original_loss instead.
Validation is 100 examples of held-out CE, not benchmark accuracy. No math benchmark evaluation has been run on this checkpoint.

Training

Table

Base	`Qwen/Qwen2.5-Math-7B` @ `b101308fe89651ea5ce025f25317fea6fc07e96e`
Objective	PSFT, clip_ratio_low 0.2, clip_ratio_high 0.28
LR / schedule	5e-5, cosine, 10% warmup (39 of 390 steps)
Optimizer	AdamW, betas (0.9, 0.95), weight decay 0.1, grad clip 1.0
Batch	256 global (8x L40S, micro-batch 4, 8 accumulation steps)
Max length	2048
Epochs / steps	1 / 390
Precision

The reference model is the same base checkpoint, frozen, pinned to the same resolved commit.

Prompt format

This model ships a chat template, which was used as-is. It injects a system instruction:

text

<|im_start|>systemPlease reason step by step, and put your final answer within \boxed{}.<|im_end|><|im_start|>user{question}<|im_end|><|im_start|>assistant{answer}<|endoftext|>

Loss is computed on answer tokens only, including the terminating EOS. The other three base models in this series ship no chat template and used a plain Question:/Answer: format, so prompt format is confounded with model when comparing across them. Within this model's PSFT/SPFT pair the format is identical.

License

Apache-2.0, inherited from the base model.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

pvdhihihi

Model Tree

Base

Qwen/Qwen2.5-Math-7B

Fine-tuned

this model

Input Modalities

Text

Output Modalities