ianlee1996

pokerbench-qwen3-14b-lora-dpo-v3

README

License: apache-2.0

Headline numbers — 4-stage evaluation

Stage 1 — Paper benchmark (regression check)

Table with columns: Metric, v2 SFT, v3 DPO, Δ
Metric	v2 SFT	v3 DPO	Δ
Overall EM	89.86%	89.91%	+0.05 ✅
Overall AA	90.32%	90.35%	+0.04 ✅
Preflop EM	92.90%	93.70%	+0.80
Postflop EM	89.56%	89.53%	-0.03

DPO doesn't break the SFT abilities — actually slightly improves preflop EM.

Stage 2 — PE 200 (production format)

Table with columns: Metric, v2 SFT, v3 DPO, Δ
Metric	v2 SFT	v3 DPO	Δ
Overall AA	84.0%	84.0%	flat
Postflop AA	77.0%	78.0%	+1.0
Parse failures	0/200	0/200	✅
Raise sizing	included	included	✅

Stage 3 — vs gpt-oss-120b (1000 hands head-to-head) ⭐

Table with columns: Variant, bb/100
Variant	bb/100
v1 SFT (paper-only)	+46.58
v1 DPO (UTG-only)	+46.46 (Δ −0.13)
v3 DPO (multi-seat + sized)	+50.92 (Δ +4.34) ✅

Beats the +3 bb/100 success threshold. This is the core RL goal — exploit non-GTO opponents.

Stage 4 — vs v2 SFT base (1000 hands self-play)

Table with columns: DPO bb/100, SFT bb/100, Delta
	DPO bb/100	SFT bb/100	Delta
v1 DPO vs v1 SFT	+22.98	+27.02	−4.03 🔴
v3 DPO vs v2 SFT	+35.12	+14.88	+20.25 ✅

DPO v3 wins against its own SFT base by +20.25 bb/100 — proves the RL loop genuinely improves on SFT.

Per-seat breakdown (v3 vs v2 SFT)

Table with columns: Seat, DPO bb/100
Seat	DPO bb/100
UTG (trained, but diluted)	-2.05 (~noise floor)
CO (key fix)	+54.33 ⭐ (was -25.87 in v1)
SB	+53.09

The CO regression that motivated this v3 retrain is fully resolved (~80 BB swing).

Method (DPO recipe)

Self-play data collection (reused from v1)

5000 hands of 6-max NLHE: ckpt-mixed × 3 seats vs openai.gpt-oss-120b-1:0 × 3 seats on AWS Bedrock
Team A (ckpt-mixed) +46.10 bb/100 baseline reproduced

Counterfactual EV preference extraction (new for v3)

Multi-seat: extract pairs for ALL 6 hero seats per hand (v1 was UTG-only)
For each seat's key hero decision: 2 candidate actions × N=10 MC rollouts × deepseek.v3.2 driving the rest of the table
Resume support across runs to avoid recomputing
Final dataset: 4686 pairs across positions:
- SB: 1655 (most postflop spots)
- UTG: 1271
- CO: 1175
- BTN: 393
- HJ: 163
- BB: 29
EV gap median 2.00 BB, p75 4.36 BB, max 75.20 BB

DPO training

TRL 1.5.1 DPOTrainer, ref_model=None (PEFT adapter-disable trick saves 28 GB)
LoRA r=32, alpha=64, target=all-linear
LR 5e-6, beta 0.1, sigmoid loss
147 steps × batch 32 = 1 epoch on 4686 pairs
Wallclock: ~25 minutes

Final metrics:

train_loss: 0.593
rewards/margins: 0.514 (v1 was 0.19)
rewards/accuracies: 73% (v1 was 64%)

Usage

python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-14B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "ianlee1996/pokerbench-qwen3-14b-lora-dpo-v3")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-14B")

system_prompt = (
    "You are a specialist in playing 6-handed No Limit Texas Holdem. "
    "Output ONLY the optimal action with no explanation. "
    "Valid formats: 'fold', 'check', 'call', 'bet N', 'raise N', 'all-in'."
)
# Inference: temp 0.1, top-p 0.95, max_tokens 16

Honest reporting

UTG seat showed -2.05 bb/100 vs the SFT base in Stage 4 — this is within statistical noise (1000 hands ≈ ±3 BB std). The same model is +19.5 bb/100 at UTG against gpt-oss-120b, so it's not a real regression. A v4 with higher UTG-data weighting is in development for users who want every seat strictly improved.
HJ/BB had relatively few training pairs (163 / 29 respectively) because GTO play makes these seats fold preflop most of the time. The eval metrics for these seats are noisier than other seats.

Reproducibility

Pipeline + experiment configs at https://github.com/IanLiYi1996/PokerBench:

bash
# 1. Self-play (~9 hours, ~$50 Bedrock)
.venv/bin/python -m scripts.collect_selfplay --config configs/eval/rl_selfplay_5k.yaml

# 2. Multi-seat counterfactual extraction (~14 hours, ~$30 Bedrock)
.venv/bin/python -m scripts.extract_preferences \
    --hand-logs data/rl/selfplay_5k.jsonl \
    --out data/rl/preferences_5k.jsonl \
    --adapter checkpoints/09_qwen3_14b_lora_mixed_v2_sized \
    --hero-seats all --n-mc 5 --max-workers 8 \
    --bedrock-model deepseek.v3.2

# 3. DPO (~25 min)
.venv/bin/python -m scripts.train --config configs/experiments/10_qwen3_14b_dpo_v3_on_v2_sft.yaml

The 4686 preference pairs and raw self-play logs are public at ianlee1996/pokerbench-rl-dpo.

Citation

bibtex
@inproceedings{zhuang2025pokerbench,
  title={PokerBench: Training Large Language Models to become Professional Poker Players},
  author={Zhuang, Richard and Gupta, Akshat and Yang, Richard and Rahane, Aniket and Li, Zhengyu and Anumanchipalli, Gopala},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  year={2025},
  url={https://arxiv.org/abs/2501.08328}
}

License

Apache-2.0, matching Qwen/Qwen3-14B and the PokerBench dataset.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider