GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
vhallac
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
vhallac
Model Tree
Qwen/Qwen3-0.6B
Input Modalities
Output Modalities
Supported Functionality
RoPE was disabled at runtime, by forcing the rotary embedding to return identity
(cos=1, sin=0). That is not expressible in a config.json, so this repo's config is a stock
Qwen3 config declaring rope_type: "default" — byte-identical to the control model's.
If you call from_pretrained and generate, RoPE will be active on a model trained without it.
You will get degenerate output, no exception, and no warning.
python
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizermodel_id = "vhallac/qwen3-0.6b-nope-recal-1b"model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16)tokenizer = AutoTokenizer.from_pretrained(model_id)# REQUIRED — force rotary embeddings to identity. Without this the model is wrong.rotary = model.model.rotary_emb_original_forward = rotary.forwarddef _identity_forward(*args, **kwargs):cos, sin = _original_forward(*args, **kwargs)return torch.ones_like(cos), torch.zeros_like(sin)rotary.forward = _identity_forwardmodel.eval()
A quick way to confirm the patch took: held-out FineWeb-Edu perplexity should be ≈ 16.9. If you see a number in the thousands, the patch did not apply.
Part of the rope-as-scaffold
research program, which tested the hypothesis that RoPE is a training scaffold that can be
discarded after pretraining.
The finding was that it cannot. This model is the evidence: it is what you get when you remove RoPE and recalibrate properly, and it is measurably worse than the same recipe with RoPE retained — across perplexity, local-order sensitivity, retrieval, and long-context behaviour.
Notably, the degradation is not because positional information vanishes. Probing shows position is still decodable and the emergent positional subspace substantially reconstructs the one RoPE supplied. The model reconstitutes the representation and still cannot do the job.
Held-out FineWeb-Edu perplexity, fp32, 5M-token frozen eval slice, identical harness for all three:
| model | CE | PPL |
|---|---|---|
Qwen/Qwen3-0.6B (base, no extra training) | 3.0819 | 21.80 |
qwen3-0.6b-rope-recal-1b (control, RoPE on) | 2.6569 | 14.25 |
| this model (RoPE removed) | 2.8260 | 16.88 |
Against the base model this looks like an improvement (21.80 → 16.88). Against the correct control it is a regression: +0.169 nats / 18.4% higher perplexity, persisting after a full recalibration. The apparent gain over base is domain adaptation; the RoPE penalty is hiding underneath it.
max_position_embeddings: 40960 inherited from the
base config. Beyond the recalibration window it degrades sharply: PPL ≈ 51 at 4096 and ≈ 298 at
8192, where the RoPE base model stays flat at ≈ 21.5.| base | Qwen/Qwen3-0.6B @ c1899de289a04d12100db370d81485cdf75e47ca |
| rotary | forced to identity for the entire run |
| corpus | HuggingFaceFW/fineweb-edu, sample-10BT, streamed in provider order |
| tokens | 1B (1907 steps × 524,288 tokens), context 2048 |
| optimizer | AdamW β=(0.9, 0.95), wd 0.1, grad clip 1.0 |
| LR | 1e-3 peak, 2% warmup, cosine → 10% of peak |
training_metrics.csv and training_manifest.json are included in this repo for full provenance.
An earlier version of this checkpoint trained at LR 3e-5 plateaued at PPL ≈ 35; that learning rate was under-tuned and the run was redone at 1e-3. This repo is the corrected run.
NOTEBOOK.md, entry 2026-07-28 — RS-amendment-2-3RS1-spec.md §10.CApache 2.0, inherited from Qwen3-0.6B.
| precision |
| bf16 |
| seed | 0 |
| hardware | 1× H100 SXM, ~7h |