Configuration
Table | |
|---|
| student | Qwen/Qwen3-4B |
| teacher | Qwen/Qwen3-32B (same tokenizer, byte-identical tokenizer.json) |
| data | SeanWang0027/polaris_hard, 15,368 competition-maths problems |
| cut | K = 4096 prefix + t = 1024 teacher tokens → max_response_length 5120 |
| schedule | asynchronous (recipe/async_rose_online), depth 3 |
| batch | train_batch_size 64, n 4 → 256 sequences per block |
| optimizer | AdamW, lr 1e-5 constant, no warmup, wd 0.01, grad clip 1.0 |
| objective | ce — masked NLL over the teacher's tokens only |
| steps | 240 (one epoch), global_step_240 |
| hardware | 8 × NVIDIA GH200 on TACC Vista — 4 teacher replicas (vLLM TP=1) + 4 trainer ranks (FSDP) |
| wall clock | 10 h 52 m (163 s/step) |
Code: cl-from-nothing/online-rose,
docs/POLARIS_QWEN3_4B_QWEN32B_K4096_8GPU.md.
Read this before using the model
The objective did not move over the 240 steps, and the weights barely changed.
These are measured, not estimated:
Table | |
|---|
actor/ce_nll_before, mean of first 20 steps | 0.6010 |
actor/ce_nll_before, mean of last 20 steps | 0.6026 (+0.3%) |
| range across all 240 steps | 0.5454 – 0.6748 (noise) |
parameters that changed at all vs Qwen/Qwen3-4B | 12.32% (543.7 M / 4.41 B) |
| global relative L2 drift vs base | 1.75e-3 |
| largest per-tensor relative drift | 3.36e-3 (layers.34.self_attn.k_proj) |
The most likely explanation is the training precision. The launcher passes
actor_rollout_ref.actor.fsdp_config.model_dtype=bfloat16, which overrides the
torch.float32 that verl otherwise forces for the actor — and verl's own comment at
fsdp_workers.py:302 says "we have to create model in fp32. Otherwise, the optimizer
is in bf16, which is incorrect." With bf16 master weights and bf16 Adam moments, one
step at lr = 1e-5 is the same order as a bf16 half-ULP at |w| ≈ 0.02, so most weights
cannot move. That 87.7% of parameters are bit-identical to the base is consistent with
this and hard to explain otherwise.
No downstream evaluation has been run. Neither this checkpoint nor the base model
has been measured on AIME 2025 or anything else. The reward logged during training
(critic/score/mean ≈ 0.06–0.14) is log-only — compute_policy_loss_ce ignores
advantages — and it scores a truncated prefix spliced to a partial continuation, so
it says nothing about the student.
Treat this as a reproduction artifact of the training run, not a model claimed to be
better than Qwen/Qwen3-4B. If you want the comparison, run both under identical
settings.
Other measurements from the run
Table with columns: metric, value, note| metric | value | note |
|---|
rose/pool_failures | 0 across all 240 steps | no teacher request ever failed |
rose/prefix_age | pinned at 2.0 = depth − 1 | the async pipeline behaved as designed |
rose/teacher_hidden_frac | ≈ 0.84 steady | 16% of teacher latency not hidden |
rose/empty | ≈ 3% mean (max 12%) |
rose/teacher_finished near zero is a property of this K/t choice, not a bug: after a
4096-token prefix, 1024 more tokens rarely reach the end of a competition-maths solution.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
mid = "SeanWang0027/student_prefix_polaris_15K_qwen3-4b_continual_Q_qwen3-32b_cutoff4096_epoch_1_mask"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, dtype="bfloat16", device_map="auto")