Overview
- Architecture: GPT-2
- Parameters: ~30M
- Training data: Self-Play w/ SF evaluation
- Task: Autoregressive chess move generation
Capabilities
ChessSLM can play chess by generating moves sequentially in SAN notation.
It has been evaluated in matches against several language models, including:
- Claude [Won against it]
- Gemini [Lost again it]
- Qwen
- GPT-2
- GPT-Neo
- Pythia
- LLaMA
- Mistral
- other small chess-oriented models
The model achieves an averaging rating of around ~1054 Elo against other language models despite its small size.
Benchmark Results
Table with columns: Model, Elo Rating| Model | Elo Rating |
|---|
| FlameF0X/ChessSLM | 1154 |
| DedeProGames/mini-chennus | 1114 |
| EleutherAI/pythia-70m-deduped | 1099 |
| nlpguy/smolchess-v2 | 1092 |
| DedeProGames/dialochess | 1078 |
| nlpguy/amdchess-v9 | 1073 |
| mlabonne/grandpythia-200k-70m | 1065 |
| FlameF0X/ChessSLM-PM | 1055 |
| DedeProGames/Chesser-248K-Mini |
Limitations
Like many language-model-based chess systems, ChessSLM has several limitations:
- Illegal move hallucinations: The model may occasionally generate moves that violate chess rules.
- No board-state verification: Moves are generated purely from learned patterns rather than a validated game state.
- Limited strategic depth: While competitive at lower Elo levels, it cannot match dedicated chess engines.
These limitations are common for pure language-model chess agents that do not use external rule engines.
Summary
ChessSLM shows that very small language models can achieve meaningful chess performance when trained on domain-specific data.
It serves as a lightweight baseline for exploring LLM-based chess agents and specialized small language models (SLMs).