XudongZhu

Qwen3-1.7B-Base-SFT-OnPolicy-BF16Delta

README

License: apache-2.0

Evaluation

Each checkpoint answers each question once (temperature 0.6, top-p 0.95, up to 15,360 tokens, seed 0; vLLM in BF16), with the prompt format below:

ID: the 500 held-out OpenR1-Math problems, scored like training (math-verify).
MATH-500: the 500 problems of MATH-500, scored the same way.
MMLU-Pro: 1,000 questions drawn with seed 0 from the test split of MMLU-Pro. The prompt asks for the letter of the answer within \boxed{}, and the answer counts as right when that letter is right.

Accuracy:

Table with columns: Step, ID, MATH-500, MMLU-Pro
Step	ID	MATH-500	MMLU-Pro
0 (base)	0.220	0.562	0.218
10	0.248	0.590	0.240
25	0.240	0.614	0.241
50	0.246	0.620	0.257
100	0.266	0.640	0.292
150	0.302	0.632	0.304
200	0.288	0.664	0.318
250	0.290	0.638	0.316
300	0.312	0.652	0.344

Checkpoints

This repository does not hold a full copy of each checkpoint. It holds step 300 as a normal checkpoint at the root, and deltas that let you rebuild the other 74 saved steps yourself. Each file in deltas/ is the XOR of a step's raw BF16 bits with those of the step before, compressed with zstd. The chain starts from the base model at revision ea980cb, which is not stored here, and manifest.json holds the SHA-256 of every delta and every step. A rebuilt step is bit-identical to the FP32 checkpoint of Qwen3-1.7B-Base-SFT-OnPolicy rounded to BF16. The low FP32 bits are not kept: for those, use that repository. Step 0 is the base model itself.

Step 300 loads directly:

python

from transformers import AutoModelForCausalLM, AutoTokenizer repo = "XudongZhu/Qwen3-1.7B-Base-SFT-OnPolicy-BF16Delta"model = AutoModelForCausalLM.from_pretrained(repo)tokenizer = AutoTokenizer.from_pretrained(repo)

For another step, rebuild it with restore.py, which needs numpy, torch, safetensors, huggingface_hub and the zstd command. It downloads the base model and walks the chain from step 1, about 6 seconds per step on a CPU:

bash

hf download XudongZhu/Qwen3-1.7B-Base-SFT-OnPolicy-BF16Delta --local-dir chainpython chain/restore.py chain --step 100 --out step100

step100 is then a normal checkpoint folder for from_pretrained. The steps are 1 to 50, then 60, 70, …, 300. --verify rebuilds every step and checks its SHA-256.

Prompt format

The user turn is the problem, a blank line, and Please reason step by step, and put your final answer within \boxed{}. There is no system prompt; tokenizer.apply_chat_template builds this:

text

<|im_start|>user{problem} Please reason step by step, and put your final answer within \boxed{}.<|im_end|><|im_start|>assistant

Training

Data. From the default split of OpenR1-Math-220k (93,733 problems):

Keep the math word problems that have a DeepSeek-R1 answer which math-verify marks correct and which is complete: 59,188 problems. (16,115 are not word problems, and 18,430 have no such answer; none is also in MATH-500.)
Shuffle them with seed 0 and hold out the first 500 for evaluation.
In this order, skip the problems whose prompt is longer than 1,024 tokens (29) or whose prompt and R1 answer together are longer than 16,384 tokens (684), and take the first 38,400.

The run trains on these 38,400 problems in this order, once. The length limit counts the R1 answer even in this run, so that both runs see the same problems.

One step. For each of 128 problems, the current model samples 8 answers (temperature 1.0, top-p 1.0, up to 15,360 tokens). math-verify compares the last 2,000 characters of each answer with the reference answer. For each problem, one correct answer is kept at random; a problem with no correct answer is skipped in this step. One optimizer step then minimizes the SFT loss on the kept answers: the mean of −log p over their tokens, without the prompt.

Table with columns: Hyperparameter, Value
Hyperparameter	Value
Optimizer	AdamW (β = 0.9, 0.999), weight decay 0, gradient clipping 1.0
Learning rate	1e-6, constant, no warmup
Batch	128 problems per step, one optimizer step
Sampling	8 answers per problem, temperature 1.0, top-p 1.0
Length	prompt ≤ 1,024 tokens, answer ≤ 15,360 tokens
Steps	300, one pass over the data
Precision	FP32 weights; BF16 compute and FP32 gradient reduction (FSDP mixed precision); BF16 sampling
Seeds	sampling 42; picking the kept answer 0

Uses and limitations

For research on training dynamics: compare weights or behavior across checkpoints, and against the off-policy twin. The run trains only on English math problems, with no instruction tuning or safety training beyond the base model; its answers are long chains of reasoning. Each accuracy above comes from one sample per question, so small differences between steps are not reliable.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

XudongZhu

Model Tree

Base

Qwen/Qwen3-1.7B-Base

Fine-tuned

this model

Input Modalities

Text

Output Modalities