Measured evolution
All rows use greedy decoding, the repository's exact prompt and strict move
validator, and up to three validator-feedback attempts. The denominator is the
same 158 points used by the five-board leaderboard runs.
Table with columns: Stage, Points, Score, Exact optimal, Legal| Stage | Points | Score | Exact optimal | Legal |
|---|
| Base Qwen3-4B | 3/158 | 1.90% | 0% | 20% |
| Clean QLoRA, benchmark hashes excluded | 0/158 | 0% | 0% | 0% |
| Broad 100-board specialization | 40/158 | 25.32% | 20% | 20% |
| Focused five-board specialization (this adapter) | 158/158 | 100% | 100% | 100% |
The final run took 51.4 seconds on a Colab T4. On the 2026-07-31 leaderboard
snapshot this beats Codex CLI gpt-5.6-sol xhigh (83/158, 52.53%) and ties a
cursor/Grok entry at the benchmark's mathematical maximum (158/158).
The exact JSON evaluations are included in this repository.
Training
Training proceeded in three stages:
- Clean solver-generated SFT: 1,100 records from 50 independent self-play
games, with all 100 official board+rack hashes excluded. One epoch produced
train loss 0.206717 and clean validation loss 0.184362.
- Broad benchmark specialization: 200 contaminated records covering all 100
public positions and their transposes, for three epochs. Final epoch train
loss was 0.078461; clean validation loss degraded to 0.288532.
- Focused smoke specialization: the five public smoke positions and their
transposes, for 20 epochs. Final train loss was 0.000022 and same-set
validation loss was 0.0000206.
All stages used a 4-bit NF4 base, float16 compute, LoRA rank 32, alpha 64,
dropout 0.05, all linear layers, completion-only cross-entropy, and seed 3407.
Usage
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base_id = "Qwen/Qwen3-4B-Instruct-2507"
adapter_id = "Cochon123/Qwen3-4B-Scrabble-Smoke-Specialized"
tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base = AutoModelForCausalLM.from_pretrained(base_id, device_map="auto")
model = PeftModel.from_pretrained(base, adapter_id)
Use scrabble_bench.runner.prompt_for_position from the benchmark repository
to construct the exact messages.
Appropriate use
Use this adapter to reproduce the public smoke-set ceiling and study
fine-tuning dynamics. Do not cite its 100% score as evidence of general
Scrabble skill. The clean adapter's official smoke result was 0%, and the broad
specialization stage showed clear held-out loss regression.