Curriculum (this run)
- Phase 1 (0-30B): 100% English
- Phase 2 (30-100B): 100% Chinese
The mirror run is Raghav-Singhal/pathlang-1p7b-runD-zh30-en70.
Architecture
- SmolLM2-1.7B backbone: 24 layers, hidden 2048, FFN 8192, 32 heads, RoPE (base 10000), RMSNorm, SwiGLU, seq len 2048
- Tokenizer: Qwen3 (multilingual, vocab 151,936); ~2.1B total params
- Converted from Megatron-LM to HF
LlamaForCausalLM
Training
- 100B tokens total, global batch size 960, 50,860 steps
- WSD LR schedule (peak 2e-4, 2000 warmup, linear decay over the final 10B tokens), bf16, Adam
- English: DCLM-edu. Chinese: FineWeb-2 hq-mmbert quality_33 (
cmn_Hani) — a larger slice than the
quality_10 used by the A/B/C runs, needed for the 70B Chinese phase.
Held-out validation loss (final checkpoint, 100B tokens)
Table with columns: English val, Chinese val | English val | Chinese val |
|---|
| this run (English-first sequential (en30->zh70)) | 3.997 | 2.531 |
In-loop 128-seq probe, cross-entropy nats/token, on this study's held-out blocks (comparable within the
D/E pair). Key finding — catastrophic forgetting under sequential training: the model ends excellent
at its second (70B) language and substantially degraded at the first, abandoned one. E.g. this run's
first-phase language rose from ~2.7 loss at the 30B switch to the value above by 100B — much sharper
forgetting than the blended A/B runs, which retained 25% of the earlier language throughout.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("Raghav-Singhal/pathlang-1p7b-runE-en30-zh70")
tok = AutoTokenizer.from_pretrained("Raghav-Singhal/pathlang-1p7b-runE-en30-zh70")
Note
License other pending confirmation; DCLM-edu and FineWeb-2 terms apply. Base (non-instruction-tuned)
research checkpoint.