ianlee1996

pokerbench-qwen3-14b-lora-dpo

README

License: apache-2.0

Three-stage training stack

This is the third LoRA layer in a stacked SFT → SFT → DPO pipeline, all trained on top of Qwen/Qwen3-14B:

Stage 1 SFT (04_qwen3_14b_lora_full_data_1epoch/checkpoint-4375): full PokerBench paper format, 1 epoch on 60k+500k samples → EM 90.07%
Stage 2 SFT (06_qwen3_14b_lora_mixed_50_50/checkpoint-250): 50/50 paper + production-PE format, 250 steps → EM 89.86%, PE AA 84.0%
Stage 3 DPO (this checkpoint): 1167 counterfactual-EV preference pairs from self-play vs gpt-oss-120b → see results below

Headline numbers (4-stage evaluation)

Table with columns: Stage, Metric, Stage 2 SFT, Stage 3 DPO, Δ, Verdict
Stage	Metric	Stage 2 SFT	Stage 3 DPO	Δ	Verdict
1 — Paper 11k	EM	89.86%	89.92%	+0.05	✅ preserved
1 — Paper 11k	AA	90.32%	90.38%	+0.06	✅ preserved
2 — PE 200	overall AA	84.0%	86.0%	+2.0	✅ improved
2 — PE 200	postflop AA	76.0%

Position-conditional finding: the per-seat breakdown of Stage 4 reveals the DPO model wins at UTG (the only position trained, +31.70 bb/100) but loses at CO (untrained, −25.87 bb/100). The LoRA's preferences don't generalize across seats — a known consequence of training only one hero position.

Stage 4 per-seat bb/100 (DPO Team A vs Stage 2 Team B)

Table with columns: Position, Team, bb/100
Position	Team	bb/100
UTG	A (DPO) ✅ trained	+31.70
HJ	B (SFT)	+1.50
CO	A (DPO) ⚠️ untrained	−25.87
BTN	B (SFT)	+6.52
SB	A (DPO) ⚠️ untrained	+63.12
BB	B (SFT)

Method (DPO recipe)

Self-play data collection

5000 hands of 6-max NLHE: ckpt-mixed × 3 seats (UTG, CO, SB) vs openai.gpt-oss-120b-1:0 × 3 seats (HJ, BTN, BB)
Captured every decision (state, action, available moves, prompt) per seat
Team A (ckpt-mixed) +46.10 bb/100 vs Team B (gpt-oss-120b), reproducing the Stage 3 baseline (±1 BB)
Wallclock: ~9 hours / Bedrock spend: ~$50

Counterfactual EV preference extraction

For each hand, pick one key hero decision (rules: river > turn > flop > preflop ≥ 2BB), generate 2 candidate actions (low-temp + high-temp + retry, with structural fallback), and run N=10 monte-carlo rollouts per candidate with deepseek.v3.2 driving every other seat to showdown. The action with higher EV becomes chosen; the other becomes rejected. Pairs with EV gap < 0.5 BB or fewer than 5 valid MC samples are dropped.

1167 pairs from 5000 hands (~23% pair rate)
EV gap median 2.10 BB, p75 4.76 BB, max 63.95 BB
All pairs are hero=UTG (paper seat 0). This is the source of the position-generalization issue noted above.
Wallclock: ~14 hours / Bedrock spend: ~$30

DPO training

TRL 1.5.1 DPOTrainer with ref_model=None (PEFT adapter-disable trick saves 28 GB ref-model memory)
LoRA r=32, alpha=64, target=all-linear (same shape as Stage 1/2)
LR 5e-6 (10× lower than SFT), beta 0.1, sigmoid loss
37 steps × batch 32 = 1 epoch on 1167 pairs
Wallclock: ~6 minutes
Final metrics: rewards/margins 0.19 (from 0.01), rewards/accuracies 64% (from 36%), train_loss 0.666

Usage

python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-14B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
# Load DPO adapter directly (it absorbs the SFT base inside)
model = PeftModel.from_pretrained(base, "ianlee1996/pokerbench-qwen3-14b-lora-dpo")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-14B")

system_prompt = (
    "You are a specialist in playing 6-handed No Limit Texas Holdem. "
    "Output ONLY the optimal action with no explanation. "
    "Valid formats: 'fold', 'check', 'call', 'bet N', 'raise N', 'all-in'."
)
messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": "<paper-format game scenario...>"},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=16, temperature=0.1, top_p=0.95, do_sample=True)
print(tokenizer.decode(out[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))

Reproducibility

Pipeline + experiment configs at https://github.com/IanLiYi1996/PokerBench:

bash
# 1. Self-play collection (~9 hours, ~$50 Bedrock)
.venv/bin/python -m scripts.collect_selfplay --config configs/eval/rl_selfplay_5k.yaml

# 2. Counterfactual extraction (~14 hours, ~$30 Bedrock)
.venv/bin/python -m scripts.extract_preferences \
    --hand-logs data/rl/selfplay_5k.jsonl \
    --out data/rl/preferences_5k.jsonl \
    --adapter checkpoints/06_qwen3_14b_lora_mixed_50_50/checkpoint-250 \
    --n-mc 10 --max-workers 8 \
    --bedrock-model deepseek.v3.2

# 3. DPO (~6 min)
.venv/bin/python -m scripts.train --config configs/experiments/07_qwen3_14b_dpo_vs_gptoss.yaml

The 1167 preference pairs and 5000 raw self-play hands used for training are public at ianlee1996/pokerbench-rl-dpo.

Honest reporting

This checkpoint is published as a research artifact, not a production-ready model. The headline takeaways:

DPO didn't break SFT: paper EM and PE AA both improved or stayed flat. The TRL DPOTrainer + PEFT adapter-disable trick is operationally sound.
DPO didn't beat SFT vs gpt-oss-120b in head-to-head: the +3 bb/100 target failed (delta −0.13 BB, statistical noise). Likely root cause: the counterfactual MC opponent (deepseek.v3.2) was different from the eval opponent (gpt-oss-120b), so DPO learned a "best response to deepseek", not a "best response to gpt-oss".
Position generalization is the bigger limitation: hero=UTG-only training caused CO position to regress by 25 BB/100. A successor checkpoint trained with hero=all-6-seats is the obvious fix.

Citation

bibtex
@inproceedings{zhuang2025pokerbench,
  title={PokerBench: Training Large Language Models to become Professional Poker Players},
  author={Zhuang, Richard and Gupta, Akshat and Yang, Richard and Rahane, Aniket and Li, Zhengyu and Anumanchipalli, Gopala},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  year={2025},
  url={https://arxiv.org/abs/2501.08328}
}

License

Apache-2.0, matching Qwen/Qwen3-14B and the PokerBench dataset.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

ianlee1996

Model Tree

Base

Qwen/Qwen3-14B

Adapter

this model

Input Modalities

Text

Output Modalities