praxis-chess

praxis-chess-reasoner-qwen3.5-4b-lora

README

License: apache-2.0

Results on the public test set

838 held-out positions from Lichess (236 puzzles, 602 mistakes from rated rapid and blitz games between 1000 and 1600 players), never seen in training. Every answer was checked claim by claim. Same base model, same data, same recipe; only the input changes from R0 to R3.

Table with columns: System, Input, Claim precision (95% CI), Chain validity (95% CI), Names the rules' cause, Invents a cause
System	Input	Claim precision (95% CI)	Chain validity (95% CI)	Names the rules' cause	Invents a cause
Rules (no model)	R3: evidence graph	100.0% (100.0–100.0)	100.0% (100.0–100.0)	100%	0%
2B LoRA	R0: board and move	34.8% (31.4–38.1)	0.5% (0.1–1.1)	1%	1%
2B LoRA	R1: + engine numbers	56.9% (54.7–58.9)	14.4% (12.1–16.8)	50%	26%
2B LoRA	R2: + flat facts	56.8% (54.6–58.8)	12.6% (10.6–14.9)	56%	27%
2B LoRA	R3: evidence graph	99.7% (99.6–99.9)	78.0% (75.4–80.8)	79%	12%
4B LoRA	R0: board and move	32.6% (29.5–35.8)	0.6% (0.1–1.2)	0%	0%
4B LoRA	R3: evidence graph	99.7% (99.5–99.9)	86.8% (84.4–89.0)	95%	16%

Claim precision: of all the claims made, the share that are true of the graph and the board.
Chain validity: the share of answers that pass every check at once.
Names the rules' cause: on the 389 positions where exactly one rule fires, how often the answer names that cause.
Invents a cause: on the 337 positions where nothing concrete happens (no material or mate changes hands within 8 plies), how often the model names a cause anyway. The right answer there is "NOT_CONCRETE, no cause".
Rules: Praxis's hand-written rules over the same graph. They always pass their own checks, and name no cause unless exactly one rule fires.

Intervals are 95% bootstrap intervals over positions (1,000 resamples, seed 0). Answers came from the q4_K_M GGUF in Ollama, greedy, with the answer JSON schema enforced. Latency: 16.1 s median per answer on a 4 GB laptop GPU (RTX 3050).

Was it good enough to ship? No, by our own rules

Before training, the study fixed four ship rules (PREREGISTRATION.md, tagged prereg-v1). They were judged on the author's own games, below.

Table with columns: Rule, Result
Rule	Result
(a) Claim precision ≥ 95%, lower bound ≥ 93%	✓ 99.5% (99.2–99.8)
(b) Not worse than the rules against fresh human labels (n ≥ 30; lower bound above −5 points)	✓ +9.1 points (+0.0 to +21.2), n = 33
(c) Invents a cause on at most 10% of nothing-concrete mistakes	✗ 15.8%
(d) Names single causes where the rules cannot (H4)	✗ not met

The Phase 8 decision was "ship the rules." In Praxis, the rules stay the diagnosis. The app uses the 2B-R3 adapter for its optional, checked commentary; this 4B is not used in the app.

Results on the author's own games (aggregate only)

The primary test set, T_C, is 890 mistakes from the author's own Chess.com games. They are private: no position, answer or label from it is published, only these totals. Prompted rows are the base model with three worked examples and no training. The prompted 4B rows cover a fixed sample of 300 positions, because of laptop time.

Table with columns: System, Items, Claim precision (95% CI), Chain validity (95% CI), Names the rules' cause, Invents a cause
System	Items	Claim precision (95% CI)	Chain validity (95% CI)	Names the rules' cause	Invents a cause
Rules (no model)	890	100.0% (100.0–100.0)	100.0% (100.0–100.0)	100%	0%
2B prompted, R0	890	7.6% (6.8–8.2)	0.1% (0.0–0.3)	22%	99%
2B prompted, R3

What the study found (2B models unless stated; on T_C, paired):

H1, supported. Training on the evidence graph (R3) beats training on the board alone (R0) by 67.6 (64.3–70.3) points of claim precision, and beats engine facts alone (R1) by 44.8 (42.6–47.1). With prompt length held level, the gain is still 48.5 (36.8–60.6) points (n = 34), so it is not just a longer prompt.
H2, supported. Training adds more where the graph is: 49.4 points of claim precision over the prompted model at R3, 24.6 at R0.
H3, supported (marginally). The 4B's edge over the 2B in claim precision is +1.8 points on the board alone and −0.2 with the graph: the evidence does the work a bigger model would. (The 4B still passes more whole chains; see the tables.)
H4, not supported. On the 194 mistakes with more than one possible cause (author's games and the public set pooled), where the rules name none, 2B-R3 names a cause in 45 answers, and 27 of those pass the checker (60%; the bar was 80%).
H5, refuted. We predicted the counterfactual (CF1) mattered most. Among 2B-R3 models trained with one component removed, the one without the threat probe (T1) names the rules' cause least often (76%, against 95–97% without any other): the threat probe carries the diagnosis.

Details: the write-up.

How to use it

The adapter reads the R3 rendering of an evidence graph: plain text with item IDs (P0, R1, CF1, T1, Δ1, …). Rendered inputs for every public test position are in the dataset (renders.R3.text). For your own games, Praxis's headless GraphCli builds the graph and its R3 rendering (--level R3) from a FEN, a move and a Stockfish binary.

Transformers + PEFT (bf16, a CUDA GPU; this is how the adapter was trained and how the quantisation check ran it):

python

import jsonimport torchfrom huggingface_hub import hf_hub_downloadfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer repo = "praxis-chess/praxis-chess-reasoner-qwen3.5-4b-lora"tok = AutoTokenizer.from_pretrained(repo, revision="v1.0")base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-4B", revision="851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",                                            dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(base, repo, revision="v1.0").eval() # One public test position, already rendered at R3.test = hf_hub_download("praxis-chess/praxis-chess-evidence-graphs", "data/test/test.jsonl",                       repo_type="dataset", revision="v1.0")item = json.loads(open(test, encoding="utf-8").readline()) messages = [{"role": "system", "content": "You are Prax. Diagnose the mistake using ONLY the evidence given. Every claim must be typed. Output JSON."},            {"role": "user", "content": item["renders"]["R3"]["text"]}]prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)out = model.generate(**ids, max_new_tokens=1000, do_sample=False)answer = tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True).strip()

The adapter was trained non-thinking: keep enable_thinking=False so the thinking block is closed before the answer starts.

Check the answer (pip install python-chess; praxis_eval is in the Praxis repo under training/src):

python

from praxis_eval import parse_diagnosis, verify report = verify(item["graph"], parse_diagnosis(answer))print(report.passed, f"{report.claims_true}/{report.claims} claims true")for v in report.violations:    print(f"rule {v.rule}: {v.message}")

Show an answer only if it passes. That is how Praxis uses this model, and it is the reason the claims are typed.

Ollama / llama.cpp: use the q4_K_M GGUF at praxis-chess/praxis-chess-reasoner-qwen3.5-4b-GGUF.

Reproduce the table: python reproduce.py run --model 4b-r3 in the Praxis repo's training/ folder, or python reproduce.py score to re-check the published answers without a GPU.

Training

Table

Base	Qwen/Qwen3.5-4B at revision `851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a` (Apache-2.0 at that revision), loaded text-only through `AutoModelForCausalLM`
Method	LoRA, r = 16, alpha = 32, dropout 0, on every attention, linear-attention and MLP projection: `q_proj`, `k_proj`, `v_proj`, `o_proj`, `in_proj_qkv`, `in_proj_z`, `in_proj_b`, `in_proj_a`, , , ,

The training targets are the rules' own diagnoses, with the prose (and, for mistakes with more than one cause, the whole answer) rewritten by a larger teacher model, Qwen3.5-27B-FP8. A teacher answer was kept only if it passed the same checker. training_config.yaml and run_manifest.json in this repo record everything needed to train it again.

Quantisation. Measured once, on the 2B-R3 adapter: on 300 public positions its q4_K_M GGUF scored −0.1 points of claim precision against bf16 (95% CI −0.5 to +0.2). The 4B's own gap was not measured.

Limitations

Tactics and material only. Version 1 explains mistakes where material or mate changes hands within 8 plies. Positional mistakes should get "NOT_CONCRETE" and no cause. It still names one on 16.3% of them in the public set.
About one answer in 8 fails a check (chain validity 86.8%). Never show an answer the checker has not passed.
True claims can still tell the wrong story. The checker catches false claims, not a misleading choice of true ones. Rules 3 and 4 cover the common cases; the rest needs human review. Human labels here are few (n = 33 fresh labels, all on the author's games).
The input format is Praxis's. It needs an evidence graph, which needs Praxis's graph builder and Stockfish.
Narrow data. Lichess puzzles and 1000–1600 rapid and blitz games, in English. Other levels and time controls are untested.
Scope of the measurement. Measured on the q4_K_M GGUF with the answer schema enforced. The bf16 adapter without a schema was measured on 300 positions only.

Files

Table with columns: File, What
File	What
`adapter_model.safetensors`, `adapter_config.json`	The LoRA adapter (PEFT 0.21.0)
`tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja`	The base tokenizer and chat template, as trained
`training_config.yaml`	The training configuration, base revision pinned
`run_manifest.json`	Provenance: base and dataset revisions, git commits, versions, file hashes, validation scores

Licence and attribution

This adapter: Apache-2.0.
Base model: Qwen/Qwen3.5-4B by the Qwen team, also Apache-2.0.
Training data: the Praxis Chess Evidence Graphs dataset, CC-BY-4.0. It is built from Lichess's CC0 puzzle and game exports.
Teacher: Qwen3.5-27B-FP8, Apache-2.0.
Engine: Stockfish (GPL-3.0) produced the evidence, run as a separate program.

Citation

bibtex

@misc{praxis_lora_v1_2026,  title  = {Does structured evidence let a small model explain chess mistakes truthfully?},  author = {Anand, Tanmay},  year   = {2026},  note   = {Praxis Chess, LoRA v1 research write-up},  url    = {https://github.com/Praxis-Chess/Praxis-Chess}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

praxis-chess

Model Tree

Base

Qwen/Qwen3.5-4B

Adapter

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Results on the public test set

Table with columns: System, Input, Claim precision (95% CI), Chain validity (95% CI), Names the rules' cause, Invents a cause
System	Input	Claim precision (95% CI)	Chain validity (95% CI)	Names the rules' cause	Invents a cause
Rules (no model)	R3: evidence graph	100.0% (100.0–100.0)	100.0% (100.0–100.0)	100%	0%
2B LoRA	R0: board and move	34.8% (31.4–38.1)	0.5% (0.1–1.1)	1%	1%
2B LoRA	R1: + engine numbers	56.9% (54.7–58.9)	14.4% (12.1–16.8)	50%	26%
2B LoRA	R2: + flat facts	56.8% (54.6–58.8)	12.6% (10.6–14.9)	56%	27%
2B LoRA	R3: evidence graph	99.7% (99.6–99.9)	78.0% (75.4–80.8)	79%	12%
4B LoRA	R0: board and move	32.6% (29.5–35.8)	0.6% (0.1–1.2)	0%	0%
4B LoRA	R3: evidence graph	99.7% (99.5–99.9)	86.8% (84.4–89.0)	95%	16%

Claim precision: of all the claims made, the share that are true of the graph and the board.
Chain validity: the share of answers that pass every check at once.
Names the rules' cause: on the 389 positions where exactly one rule fires, how often the answer names that cause.
Invents a cause: on the 337 positions where nothing concrete happens (no material or mate changes hands within 8 plies), how often the model names a cause anyway. The right answer there is "NOT_CONCRETE, no cause".
Rules: Praxis's hand-written rules over the same graph. They always pass their own checks, and name no cause unless exactly one rule fires.

Was it good enough to ship? No, by our own rules

Before training, the study fixed four ship rules (PREREGISTRATION.md, tagged prereg-v1). They were judged on the author's own games, below.

Table with columns: Rule, Result
Rule	Result
(a) Claim precision ≥ 95%, lower bound ≥ 93%	✓ 99.5% (99.2–99.8)
(b) Not worse than the rules against fresh human labels (n ≥ 30; lower bound above −5 points)	✓ +9.1 points (+0.0 to +21.2), n = 33
(c) Invents a cause on at most 10% of nothing-concrete mistakes	✗ 15.8%
(d) Names single causes where the rules cannot (H4)	✗ not met

The Phase 8 decision was "ship the rules." In Praxis, the rules stay the diagnosis. The app uses the 2B-R3 adapter for its optional, checked commentary; this 4B is not used in the app.

Results on the author's own games (aggregate only)

Table with columns: System, Items, Claim precision (95% CI), Chain validity (95% CI), Names the rules' cause, Invents a cause
System	Items	Claim precision (95% CI)	Chain validity (95% CI)	Names the rules' cause	Invents a cause
Rules (no model)	890	100.0% (100.0–100.0)	100.0% (100.0–100.0)	100%	0%
2B prompted, R0	890	7.6% (6.8–8.2)	0.1% (0.0–0.3)	22%	99%
2B prompted, R3

What the study found (2B models unless stated; on T_C, paired):

H1, supported. Training on the evidence graph (R3) beats training on the board alone (R0) by 67.6 (64.3–70.3) points of claim precision, and beats engine facts alone (R1) by 44.8 (42.6–47.1). With prompt length held level, the gain is still 48.5 (36.8–60.6) points (n = 34), so it is not just a longer prompt.
H2, supported. Training adds more where the graph is: 49.4 points of claim precision over the prompted model at R3, 24.6 at R0.
H3, supported (marginally). The 4B's edge over the 2B in claim precision is +1.8 points on the board alone and −0.2 with the graph: the evidence does the work a bigger model would. (The 4B still passes more whole chains; see the tables.)
H4, not supported. On the 194 mistakes with more than one possible cause (author's games and the public set pooled), where the rules name none, 2B-R3 names a cause in 45 answers, and 27 of those pass the checker (60%; the bar was 80%).
H5, refuted. We predicted the counterfactual (CF1) mattered most. Among 2B-R3 models trained with one component removed, the one without the threat probe (T1) names the rules' cause least often (76%, against 95–97% without any other): the threat probe carries the diagnosis.

Details: the write-up.

How to use it

Transformers + PEFT (bf16, a CUDA GPU; this is how the adapter was trained and how the quantisation check ran it):

python

import jsonimport torchfrom huggingface_hub import hf_hub_downloadfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer repo = "praxis-chess/praxis-chess-reasoner-qwen3.5-4b-lora"tok = AutoTokenizer.from_pretrained(repo, revision="v1.0")base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-4B", revision="851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",                                            dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(base, repo, revision="v1.0").eval() # One public test position, already rendered at R3.test = hf_hub_download("praxis-chess/praxis-chess-evidence-graphs", "data/test/test.jsonl",                       repo_type="dataset", revision="v1.0")item = json.loads(open(test, encoding="utf-8").readline()) messages = [{"role": "system", "content": "You are Prax. Diagnose the mistake using ONLY the evidence given. Every claim must be typed. Output JSON."},            {"role": "user", "content": item["renders"]["R3"]["text"]}]prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)out = model.generate(**ids, max_new_tokens=1000, do_sample=False)answer = tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True).strip()

The adapter was trained non-thinking: keep enable_thinking=False so the thinking block is closed before the answer starts.

Check the answer (pip install python-chess; praxis_eval is in the Praxis repo under training/src):

python

from praxis_eval import parse_diagnosis, verify report = verify(item["graph"], parse_diagnosis(answer))print(report.passed, f"{report.claims_true}/{report.claims} claims true")for v in report.violations:    print(f"rule {v.rule}: {v.message}")

Show an answer only if it passes. That is how Praxis uses this model, and it is the reason the claims are typed.

Ollama / llama.cpp: use the q4_K_M GGUF at praxis-chess/praxis-chess-reasoner-qwen3.5-4b-GGUF.

Reproduce the table: python reproduce.py run --model 4b-r3 in the Praxis repo's training/ folder, or python reproduce.py score to re-check the published answers without a GPU.

Training

Table

Base	Qwen/Qwen3.5-4B at revision `851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a` (Apache-2.0 at that revision), loaded text-only through `AutoModelForCausalLM`
Method	LoRA, r = 16, alpha = 32, dropout 0, on every attention, linear-attention and MLP projection: `q_proj`, `k_proj`, `v_proj`, `o_proj`, `in_proj_qkv`, `in_proj_z`, `in_proj_b`, `in_proj_a`, , , ,

Limitations

Tactics and material only. Version 1 explains mistakes where material or mate changes hands within 8 plies. Positional mistakes should get "NOT_CONCRETE" and no cause. It still names one on 16.3% of them in the public set.
About one answer in 8 fails a check (chain validity 86.8%). Never show an answer the checker has not passed.
True claims can still tell the wrong story. The checker catches false claims, not a misleading choice of true ones. Rules 3 and 4 cover the common cases; the rest needs human review. Human labels here are few (n = 33 fresh labels, all on the author's games).
The input format is Praxis's. It needs an evidence graph, which needs Praxis's graph builder and Stockfish.
Narrow data. Lichess puzzles and 1000–1600 rapid and blitz games, in English. Other levels and time controls are untested.
Scope of the measurement. Measured on the q4_K_M GGUF with the answer schema enforced. The bf16 adapter without a schema was measured on 300 positions only.

Files

Table with columns: File, What
File	What
`adapter_model.safetensors`, `adapter_config.json`	The LoRA adapter (PEFT 0.21.0)
`tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja`	The base tokenizer and chat template, as trained
`training_config.yaml`	The training configuration, base revision pinned
`run_manifest.json`	Provenance: base and dataset revisions, git commits, versions, file hashes, validation scores

Licence and attribution

This adapter: Apache-2.0.
Base model: Qwen/Qwen3.5-4B by the Qwen team, also Apache-2.0.
Training data: the Praxis Chess Evidence Graphs dataset, CC-BY-4.0. It is built from Lichess's CC0 puzzle and game exports.
Teacher: Qwen3.5-27B-FP8, Apache-2.0.
Engine: Stockfish (GPL-3.0) produced the evidence, run as a separate program.

Citation

bibtex

@misc{praxis_lora_v1_2026,  title  = {Does structured evidence let a small model explain chess mistakes truthfully?},  author = {Anand, Tanmay},  year   = {2026},  note   = {Praxis Chess, LoRA v1 research write-up},  url    = {https://github.com/Praxis-Chess/Praxis-Chess}}