GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
robinzixuan
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
robinzixuan
Model Tree
Qwen/Qwen3-0.6B
Input Modalities
Output Modalities
Supported Functionality
| Base model | Qwen/Qwen3-0.6B |
| Variant | Vanilla AttnResidual baseline |
| Token-level attention | Standard softmax |
| Depth routing | AttentionResidual |
| Null → depth coupling | — |
| Training | See the paper for the training setup |
| Parameters | 0.63B (F32) |
| Config class | Qwen3ForCausalLM |
| License | apache-2.0 (inherited from the base model) |
AttnResidual architectures add a depth-wise normalization channel that improves inter-layer routing flexibility, but it also amplifies attention sinks, activation outliers, and the resulting loss of inference stability and quantization robustness. OASIS adds a Softmax1-based null space and couples token-level null evidence to depth routing through an inter-layer null signal, reducing sink-dominated routing. This repo is one arm of that comparison (OASIS, its ablations, and sink / vanilla baselines); see the table at the bottom for all variants.
⚠️ Weights only. The checkpoint contains extra AttentionResidual (and, where applicable, Softmax1 / sink) parameters that are not part of the stock
Qwen3ForCausalLM. Loading it with plainAutoModelForCausalLMsilently drops those parameters and runs standard attention, so outputs will not reflect this variant. Build the model with the OASIS modeling code, then load the weights.
python
import glob, os, torchfrom huggingface_hub import snapshot_downloadfrom safetensors.torch import load_filefrom transformers import AutoTokenizer model_id = "robinzixuan/qwen3-0.6b-vanilla"path = snapshot_download(model_id)tokenizer = AutoTokenizer.from_pretrained(path) state_dict = {}for f in sorted(glob.glob(os.path.join(path, "*.safetensors"))): state_dict.update(load_file(f)) # TODO: replace with the model constructor from the OASIS codebase (variant="vanilla")model = build_oasis_model(path, variant="vanilla")missing, unexpected = model.load_state_dict(state_dict, strict=False)assert not missing and not unexpected, (missing, unexpected)model = model.to("cuda" if torch.cuda.is_available() else "cpu").eval() inputs = tokenizer("Hello, ", return_tensors="pt").to(model.device)outputs = model.generate(**inputs, max_new_tokens=64)print(tokenizer.decode(outputs[0], skip_special_tokens=True))To see which parameters stock transformers would drop:
python
from transformers import AutoModelForCausalLM _, info = AutoModelForCausalLM.from_pretrained("robinzixuan/qwen3-0.6b-vanilla", output_loading_info=True)print(info["unexpected_keys"][:10]) # AttentionResidual / sink parameters ignored by Qwen3ForCausalLMSee the paper for the attention-sink, outlier (‖·‖∞, kurtosis) and post-quantization (W8A8 / W4A4) comparisons across variants.
config.json declares Qwen3ForCausalLM with extra AttentionResidual modules; requires the OASIS modeling code (see Usage).| Model | Base | Variant | Context |
|---|---|---|---|
| Phi4_attn_residual | Phi-4-mini-instruct | Vanilla AttnResidual baseline | standard |
| Phi4_attn_OASIS | Phi-4-mini-instruct | OASIS | standard |
| Phi4_attn_AoS | Phi-4-mini-instruct | AoS (attention-only Softmax1) | standard |
| qwen3-0.6b-oasis | Qwen3-0.6B | OASIS | standard |
| qwen3-0.6b-vanilla (this model) | Qwen3-0.6B | Vanilla AttnResidual baseline | standard |
| llama-3.2-1b-vanilla | Llama-3.2-1B | Vanilla AttnResidual baseline | standard |
| llama-3.2-1b-oasis | Llama-3.2-1B | OASIS | standard |
| oasis_llama3.2_1b_6k | Llama-3.2-1B | OASIS | 6k |
| vanilla_llama3.2_1b_6k | Llama-3.2-1B | Vanilla AttnResidual baseline | 6k |
| oasis_qwen3_0.6b_6k | Qwen3-0.6B | OASIS | 6k |
| vanilla_qwen3_0.6b_6k | Qwen3-0.6B | Vanilla AttnResidual baseline | 6k |
| sink_logit_llama3.2_1b | Llama-3.2-1B | Sink baseline S1 (sink logit) | 6k |
| sink_token_llama3.2_1b | Llama-3.2-1B | Sink baseline S2 (prefix sink) | 6k |
| s0_softmax1_noroute_llama3.2_1b | Llama-3.2-1B | Ablation S0 (Softmax1, no routing) | 6k |
bibtex
@article{luo2026oasis, title = {Attention Sinks and Outliers in Attention Residuals}, author = {Luo, Haozheng and Dai, Haoran and Zhang, Shaoyang and Chen, Xi and Jiang, Eric Hanchen and Li, Yijiang and Huang, Jingyuan and Qiu, Chenghao and Xu, Chenwei and Pan, Zhenyu and Zhang, Haotian and Wang, Binghui and Chen, Yan}, journal = {arXiv preprint arXiv:2605.17887}, year = {2026}}import glob, os, torchfrom huggingface_hub import snapshot_downloadfrom safetensors.torch import load_filefrom transformers import AutoTokenizer model_id = "robinzixuan/qwen3-0.6b-vanilla"path = snapshot_download(model_id)tokenizer = AutoTokenizer.from_pretrained(path) state_dict = {}for f in sorted(glob.glob(os.path.join(path, "*.safetensors"))): state_dict.update(load_file(f)) # TODO: replace with the model constructor from the OASIS codebase (variant="vanilla")model = build_oasis_model(path, variant="vanilla")missing, unexpected = model.load_state_dict(state_dict, strict=False)assert not missing and not unexpected, (missing, unexpected)model = model.to("cuda" if torch.cuda.is_available() else "cpu").eval() inputs = tokenizer("Hello, ", return_tensors="pt").to(model.device)outputs = model.generate(**inputs, max_new_tokens=64)print(tokenizer.decode(outputs[0], skip_special_tokens=True))from transformers import AutoModelForCausalLM _, info = AutoModelForCausalLM.from_pretrained("robinzixuan/qwen3-0.6b-vanilla", output_loading_info=True)print(info["unexpected_keys"][:10]) # AttentionResidual / sink parameters ignored by Qwen3ForCausalLM@article{luo2026oasis, title = {Attention Sinks and Outliers in Attention Residuals}, author = {Luo, Haozheng and Dai, Haoran and Zhang, Shaoyang and Chen, Xi and Jiang, Eric Hanchen and Li, Yijiang and Huang, Jingyuan and Qiu, Chenghao and Xu, Chenwei and Pan, Zhenyu and Zhang, Haotian and Wang, Binghui and Chen, Yan}, journal = {arXiv preprint arXiv:2605.17887}, year = {2026}}