Before vs after, on held-out math problems
The 15 problems below were never in training, in any language. There is no language instruction in the prompt (Benak condition C0), and each problem is
sampled once (temperature 0.6) per language. Reasoning language is measured by GlotLID on the <think> block, after removing sentences that quote the prompt.
Table with columns: language, English share of thinking, in-language share of thinking, accuracy| language | English share of thinking | in-language share of thinking | accuracy |
|---|
| Indonesian | 97% → 7% | 2% → 84% | 20.0% → 26.7% |
| Javanese | 92% → 0% | 6% → 58% | 6.7% → 6.7% |
| Sundanese | 94% → 1% | 5% → 19% | 6.7% → 6.7% |
| Thai | 97% → 0% | 2% → 100% | 0.0% → 20.0% |
| all | 95% → 2% | 4% → 66% | 8.3% → 15.0% |
Training
- 982 examples, 1 epoch, LoRA r=16 on all attention and MLP projections, lr 0.0002, max length 768, effective batch 8.
- Trained on one GTX 1660 SUPER (6 GB) with fp16 autocast. The final logged loss was 1.5314, after 4723 s.
- Code:
distill.py in the dataset repo. Held-out ids: [1, 5, 6, 7, 8, 14, 15, 17, 40, 43, 47, 50, 55, 61, 66].
Statistics
-
The English share of thinking fell in 58 of 58 paired items and rose in 0 (exact sign test p = 6.9e-18).
-
The accuracy change is not significant (McNemar 3 vs 7, p = 0.3438).
-
Cost: 9/60 outputs fall into repetition loops (> 20% of the thinking is one repeated 8-gram), against 0/60 before. One epoch of LoRA on about 1k traces makes the model think in the language but also makes it loop. That is a measured trade-off, not hidden.
-
Fix for the loops, measured: decoding with repetition_penalty=1.1 (no retraining) cuts loops from 9/60 to 1/60. Finished thinking rises 45% → 80.0%, accuracy is 21.7%, the English share of thinking is 9%, and the in-language share is 63%. Recommended: model.generate(..., repetition_penalty=1.1).
Limits
15 problems × 4 languages = 60 generations per side, one sample each. Accuracy at 0.6B is low on these multi-step problems, and the
claim here is about the language of thinking, not about math skill. Sundanese moved least (in-language 5% → 19%).