vhallac

qwen3-0.6b-rope-recal-1b

README

License: apache-2.0

Why a control model exists

Part of the rope-as-scaffold research program, which tested whether RoPE is a training scaffold that can be discarded after pretraining.

The problem this checkpoint solves: earlier comparisons pitted a RoPE-removed model that had received 1B extra tokens against an untouched base model that had received none. Any difference confounds "RoPE was removed" with "received extra in-domain training." Since the recalibration corpus is also the evaluation domain, that confound is large and it runs in a direction that flatters the RoPE-removed model.

This model holds the training constant so the comparison isolates RoPE. It changed the program's conclusions: one prior result reversed sign, another was halved in magnitude, and a persistent perplexity penalty became visible that had been invisible against the un-adapted baseline.

Results

Held-out FineWeb-Edu perplexity, fp32, 5M-token frozen eval slice, identical harness for all three:

Table with columns: model, CE, PPL
model	CE	PPL
`Qwen/Qwen3-0.6B` (base, no extra training)	3.0819	21.80
this model (control, RoPE on)	2.6569	14.25
`qwen3-0.6b-nope-recal-1b` (RoPE removed)	2.8260	16.88

Reading these correctly matters. Recalibration is worth 0.425 nats (3.0819 → 2.6569). Removing RoPE gives 0.169 nats of that back (2.6569 → 2.8260). The RoPE-removed model's apparent improvement over base (21.80 → 16.88) is the net of those two, and reporting it against base alone makes a real regression look like a gain.

Known limitations

Effective context ≈ 2048 tokens for the recalibrated behaviour, despite max_position_embeddings: 40960 inherited from base. Unlike its NoPE sibling this model does not collapse beyond 2048 (it inherits base's 32k-context pretraining), but its recalibration only covered 2048.
English-only recalibration corpus (FineWeb-Edu), while Qwen3 is multilingual + code + math. Expect domain skew relative to base outside the eval domain.
Not instruction-tuned beyond whatever base carried; recalibration was plain LM training.
Single seed, single recipe, 0.6B scale.

Training

Table

base	`Qwen/Qwen3-0.6B` @ `c1899de289a04d12100db370d81485cdf75e47ca`
rotary	active throughout (no patch applied)
corpus	`HuggingFaceFW/fineweb-edu`, `sample-10BT`, streamed in provider order
tokens	1B (1907 steps × 524,288 tokens), context 2048
optimizer	AdamW β=(0.9, 0.95), wd 0.1, grad clip 1.0
LR	1e-3 peak, 2% warmup, cosine → 10% of peak

Trained from the same cached token file as its NoPE sibling, not merely the same corpus — the two models saw the identical token sequence in the identical order. training_metrics.csv and training_manifest.json are included for full provenance.

Note: training_manifest.json in this repo carries "artifact": "qwen3-droped" — a known mislabel from a hardcoded string in the training script. Every functional field is correct for this model, including rotary_patch_applied: false and output_dir: /workspace/qwen3-rope-recal.

Reproduction and full analysis

Code, specs, and lab notebook: vhallac/crockpot-experiments
The controlled comparison this model enabled: NOTEBOOK.md, entry 2026-07-28 — RS-amendment-2-3
Pre-registered plan: RS-amendment-2-3.md; recipe of record RS1-spec.md §11

License

Apache 2.0, inherited from Qwen3-0.6B.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

vhallac

Model Tree

Base

Qwen/Qwen3-0.6B

Fine-tuned

this model

Input Modalities

Text

Output Modalities