What this is
Qwen3-1.7B-Base, continued-pretrained for about 20B tokens (one pass over a web / PDF / code /
agent-trajectory mix, 16K-token sequences, AdamW, peak LR 3e-4 with cosine decay). This is a
standard Qwen3 model (loads without trust_remote_code). It is the control for the
looped models in the Depth Recurrent Vector collection: same data, same recipe,
no looping. Base model before continued pretraining: test loss 1.5515.
Test loss
1.1460 on the held-out test set (lower is better).
Averaged-weights version (ema branch)
The ema branch of this repo holds a running average of the weights over training, which is
usually slightly better than the final weights. Recent steps count more than early ones (power
EMA with gamma = 36.7: step t gets weight proportional to t^36.7, so the average effectively
covers the last few percent of training). For models that average several runs, the ema
branch averages those runs' averaged weights. Same architecture and code, only the weights differ:
model = AutoModelForCausalLM.from_pretrained("CharlesDDDD/DRV-Qwen3-1.7B-midtrain-noloop", revision="ema", dtype="bfloat16", trust_remote_code=True)
Test loss: 1.1460 (final weights, main) vs 1.1426 (averaged, ema).