Model Details
- Developed by: Diginyx
- Base model: Qwen/Qwen3.5-27B
- Model type: Causal LM — LoRA adapter (PEFT)
- Language: English
- License: Apache 2.0
- Fine-tuning method: QLoRA (4-bit NF4 base + LoRA adapters)
- Training framework: TRL + HuggingFace PEFT
Training Methodology
The model is trained via supervised fine-tuning on game transcripts where the outcome was a win (positive clemscore contribution), filtered from rollouts of the base Qwen3.5-27B-Instruct model playing all clembench 2.0 games. The goal is to teach the policy the turn-level response patterns associated with successful multi-player game trajectories.
Training pipeline:
- Run the base model on all clembench games to collect rollout transcripts
- Filter to transcripts with a positive game outcome (win)
- Fine-tune on the winning turns using QLoRA
Design decisions:
- Positive-only filtering: Using only winning trajectories (rather than all rollouts with reward labels) avoids the model learning from ambiguous partial-credit or losing sequences, keeping the training signal clean.
- 4-bit QLoRA: Reduces VRAM from ~55 GB to ~14 GB, allowing the full 27B model to train on a single 48 GB A40 alongside optimizer states. This makes training accessible without multi-node tensor parallelism.
- LoRA over full fine-tune: Preserves the base model's general language capabilities while adapting the turn-level game response style. The small adapter (r=16) also prevents overfitting on the filtered game corpus.
- Max length 1024: Game turns are typically short; truncating at 1024 tokens keeps the full-vocabulary causal-LM logits tensor (batch × seq × 152k vocab) within GPU memory budget.
Training Data
- Dataset: colab-potsdam/playpen-data — clembench 2.0 game instances (training split)
- Games: All games present in the benchmark (wordle, taboo, reference, clean_up, and others)
- Filtering: Turn-level transcripts where the final game outcome was a win
- Preprocessing: Chat-templated using Qwen3.5 instruction template with
enable_thinking=False
Hyperparameters
Table with columns: Parameter, Value| Parameter | Value |
|---|
| Learning rate | 2e-4 |
| LR scheduler | Cosine with warmup |
| Epochs trained | 1 (early stopping on val loss) |
| Per-device batch size | 4 |
| Effective batch size | 128 (auto grad-accum across GPUs) |
| Max sequence length | 1024 tokens |
| LoRA rank (r) | 16 |
| LoRA alpha | 32 |
| LoRA dropout |
Compute
Table with columns: Resource, Details| Resource | Details |
|---|
| Hardware | 4× NVIDIA A40 (48 GB) |
| Cluster | University of Michigan HPC (SLURM) |
| Training time | ~1 epoch over the filtered positive corpus |
| Total FLOPs | ~2.64 × 10¹⁸ |
Evaluation
Evaluated on the Playpen benchmark (clembench 2.0) using clemscore (quality-weighted success rate across all games) and statscore (static benchmark aggregate).
When used with the companion PRM (Diginyx/Qwen3.5-27B-prm-ep1) in a best-of-N or beam search guided inference setup, this model achieves higher clemscore than the greedy baseline.
Usage
Standalone (greedy inference)
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type="nf4",
)
base = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen3.5-27B",
quantization_config=bnb_config,
device_map="auto",
)
model = PeftModel.from_pretrained(base, "Diginyx/Qwen3.5-27B-sft-ep1")
tokenizer = AutoTokenizer.from_pretrained("Diginyx/Qwen3.5-27B-sft-ep1")
With PRM-guided inference (best-of-N)
Install Playpen and register the model in model_registry.json:
{
"model_name": "Qwen3.5-27B-sft-ep1",
"backend": "huggingface_local",
"huggingface_id": "Qwen/Qwen3.5-27B",
"model_config": {
"premade_chat_template": true,
"load_in_4bit": true,
"chat_template_kwargs": {"enable_thinking": false},
"peft_model": "Diginyx/Qwen3.5-27B-sft-ep1"
}
}
Then run:
python examples/trl/prm_eval.py \
--policy-model Qwen3.5-27B-sft-ep1 \
--prm-path Diginyx/Qwen3.5-27B-prm-ep1 \
--game-all \
--n-candidates 4 \
--temperature 0.7 \
--max-tokens 2048
With beam search
python examples/trl/prm_eval.py \
--policy-model Qwen3.5-27B-sft-ep1 \
--prm-path Diginyx/Qwen3.5-27B-prm-ep1 \
--mode beam-search \
--n-candidates 4 \
--num-beam-iterations 20 \
--game-all \
--temperature 0.7 \
--max-tokens 2048
Code
The full training and inference pipeline is available at Diginyx/playpen-prm-code:
Table with columns: Script, Purpose| Script | Purpose |
|---|
prm_trainer.py | PRM rollout collection + training |
prm_eval.py | Best-of-N and beam search guided inference |
sft_trainer_lora.py | SFT policy fine-tuning |
eval_validation_one.py | Plain (no PRM) evaluation |
run_sft.sh | SLURM job for SFT training |
run_27b_prm_eval.sh |
Companion Models
Framework Versions
- PEFT 0.19.1
- TRL
- Transformers
- bitsandbytes