XudongZhu

Qwen3-1.7B-Base-SFT-OffPolicy

README

License: apache-2.0

Checkpoints

Each checkpoint is a branch named after its step: step1 to step50, then step60, step70, …, step300. main holds the last one (step 300). Step 0 is the base model itself.

python

from transformers import AutoModelForCausalLM, AutoTokenizer repo = "XudongZhu/Qwen3-1.7B-Base-SFT-OffPolicy"model = AutoModelForCausalLM.from_pretrained(repo, revision="step100")tokenizer = AutoTokenizer.from_pretrained(repo, revision="step100")

Without revision, you get step 300. All branches: huggingface_hub.list_repo_refs(repo).branches.

Prompt format

The user turn is the problem, a blank line, and Please reason step by step, and put your final answer within \boxed{}. There is no system prompt; tokenizer.apply_chat_template builds this:

text

<|im_start|>user{problem} Please reason step by step, and put your final answer within \boxed{}.<|im_end|><|im_start|>assistant

Training

Data. From the default split of OpenR1-Math-220k (93,733 problems):

Keep the math word problems that have a DeepSeek-R1 answer which math-verify marks correct and which is complete: 59,188 problems. (16,115 are not word problems, and 18,430 have no such answer; none is also in MATH-500.)
Shuffle them with seed 0 and hold out the first 500 for evaluation.
In this order, skip the problems whose prompt is longer than 1,024 tokens (29) or whose prompt and R1 answer together are longer than 16,384 tokens (684), and take the first 38,400.

The run trains on these 38,400 problems in this order, once, each with its R1 answer from step 1 (one picked at random, with seed 0, when there are several).

One step. One optimizer step on the R1 answers of 128 problems, with the SFT loss: the mean of −log p over the answer tokens, without the prompt.

Table with columns: Hyperparameter, Value
Hyperparameter	Value
Optimizer	AdamW (β = 0.9, 0.999), weight decay 0, gradient clipping 1.0
Learning rate	1e-6, constant, no warmup
Batch	128 problems with their R1 answers per step, one optimizer step
Length	prompt and answer ≤ 16,384 tokens
Steps	300, one pass over the data
Precision	FP32 weights; BF16 compute and FP32 gradient reduction (FSDP mixed precision)
Seeds	0 (the R1 answer picked per problem)

Uses and limitations

For research on training dynamics: compare weights or behavior across checkpoints, and against the on-policy twin. The run trains only on English math problems, with no instruction tuning or safety training beyond the base model; its answers are long chains of reasoning.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

XudongZhu

Model Tree

Base

Qwen/Qwen3-1.7B-Base

Fine-tuned

this model

Input Modalities

Text

Output Modalities