Training
- Architecture: Qwen3-0.6B (28 layers, d=1024, 16 q / 8 kv heads, head_dim 128, SwiGLU 3072,
QK-norm, RoPE theta 1e6, tied embeddings). Vocab = len(tokenizer) = 151669.
- Data: DCLM only, documents separated by
<|endoftext|>, packed to 4096 tokens with
document masking (no cross-document attention).
- 19,074 steps x 256 sequences x 4096 tokens = 20.0B tokens; AdamW (lr 2.65e-3, betas 0.9/0.95,
wd 0.05), 954 warmup steps, cosine decay to 1% of peak; bf16; 32 H200 GPUs.
Evaluation (final checkpoint)
OLMES (v0.4.13 port), base-model formats:
Table with columns: task, metric, score| task | metric | score |
|---|
| SciQ (RC, 5-shot) | acc | 0.798 |
| ARC-Easy (RC, 5-shot) | acc | 0.688 |
| MMLU (RC, 5-shot, macro over 57 subjects) | acc | 0.336 |
| GSM8K (8-shot CoT) | exact match | 0.020 |
| TriviaQA (5-shot) | F1 | 0.254 |
| GSM8K | gold-answer bits/byte | 0.666 |
| MBPP (3-shot) | gold-solution bits/byte | 0.912 |
Held-out NLL per token on Dolmino sources (1000 docs each): dclm 2.610, math 1.632, flan 3.517,
wiki 2.645, pes2o 2.763, stackexchange 2.773.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizertok = AutoTokenizer.from_pretrained("scottgeng00/qwen3-0.6b-dclm-20b")model = AutoModelForCausalLM.from_pretrained("scottgeng00/qwen3-0.6b-dclm-20b", torch_dtype="bfloat16")