Stage
S1 — S0 plus a parallel-data anneal. Branched from S0 at step 40,000 and
ran the decay window (40,000 -> 45,149) on 1.2B tokens containing 30%
Chinese-English parallel text.
S0 and S1 are a controlled pair: same starting checkpoint, same
hyperparameters, same learning-rate schedule. Only the data differs.
What to expect
This is a 13B-token model. For scale: Qwen3-0.6B-Base saw 36T tokens —
about 2,700x more. Treat the numbers below as what that budget buys, not as a
competitive result.
Table with columns: WMT22 5-shot, BLEU, COMET| WMT22 5-shot | BLEU | COMET |
|---|
| zh->en | 8.99 | 0.6855 |
| en->zh | 27.29 | 0.7743 |
Against S0 (0.54 / 3.97) this is more than an order of magnitude, and the
jump is qualitative: S0 ignores the examples and gets the output language
wrong, S1 actually translates. 1.2B tokens of anneal data — 30% of it
parallel — is what made in-context learning appear.
Tokenizer
The tokenizer is a compiled C++ extension, not loadable by
AutoTokenizer. The release ships tokenizer.py and example_load.py:
pip install git+https://github.com/Ismantic/PieceTokenizer
python example_load.py
The model code (model.py, checkpoint.py) is bundled too — the package
depends only on torch plus the tokenizer extension, not on transformers.
Training
Full pipeline, data mixes and every design decision (including the mistakes)
are documented in https://github.com/Ismantic/Summer. Notably
docs/WHY.md records why fp32 master weights are mandatory, why the learning
rate schedule is WSD rather than cosine, and what the vocabulary swap cost.
License
Apache-2.0. Training corpora are public datasets (FineWeb-Edu, Cosmopedia,
CCI3-HQ, SkyPile, WMT19, OPUS-100 and others; see data/source.py upstream).
Please observe their respective licenses.