Qwen3.5-27B-sft-ep1 API & Inference Endpoint

Model Details

Developed by: Diginyx
Base model: Qwen/Qwen3.5-27B
Model type: Causal LM — LoRA adapter (PEFT)
Language: English
License: Apache 2.0
Fine-tuning method: QLoRA (4-bit NF4 base + LoRA adapters)
Training framework: TRL + HuggingFace PEFT

Training Methodology

The model is trained via supervised fine-tuning on game transcripts where the outcome was a win (positive clemscore contribution), filtered from rollouts of the base Qwen3.5-27B-Instruct model playing all clembench 2.0 games. The goal is to teach the policy the turn-level response patterns associated with successful multi-player game trajectories.

Training pipeline:

Run the base model on all clembench games to collect rollout transcripts
Filter to transcripts with a positive game outcome (win)
Fine-tune on the winning turns using QLoRA

Design decisions:

Positive-only filtering: Using only winning trajectories (rather than all rollouts with reward labels) avoids the model learning from ambiguous partial-credit or losing sequences, keeping the training signal clean.
4-bit QLoRA: Reduces VRAM from ~55 GB to ~14 GB, allowing the full 27B model to train on a single 48 GB A40 alongside optimizer states. This makes training accessible without multi-node tensor parallelism.
LoRA over full fine-tune: Preserves the base model's general language capabilities while adapting the turn-level game response style. The small adapter (r=16) also prevents overfitting on the filtered game corpus.
Max length 1024: Game turns are typically short; truncating at 1024 tokens keeps the full-vocabulary causal-LM logits tensor (batch × seq × 152k vocab) within GPU memory budget.

Training Data

Dataset: colab-potsdam/playpen-data — clembench 2.0 game instances (training split)
Games: All games present in the benchmark (wordle, taboo, reference, clean_up, and others)
Filtering: Turn-level transcripts where the final game outcome was a win
Preprocessing: Chat-templated using Qwen3.5 instruction template with enable_thinking=False

Hyperparameters

Table with columns: Parameter, Value
Parameter	Value
Learning rate	2e-4
LR scheduler	Cosine with warmup
Epochs trained	1 (early stopping on val loss)
Per-device batch size	4
Effective batch size	128 (auto grad-accum across GPUs)
Max sequence length	1024 tokens
LoRA rank (r)	16
LoRA alpha	32
LoRA dropout

Compute

Table with columns: Resource, Details
Resource	Details
Hardware	4× NVIDIA A40 (48 GB)
Cluster	University of Michigan HPC (SLURM)
Training time	~1 epoch over the filtered positive corpus
Total FLOPs	~2.64 × 10¹⁸

Evaluation

Evaluated on the Playpen benchmark (clembench 2.0) using clemscore (quality-weighted success rate across all games) and statscore (static benchmark aggregate).

When used with the companion PRM (Diginyx/Qwen3.5-27B-prm-ep1) in a best-of-N or beam search guided inference setup, this model achieves higher clemscore than the greedy baseline.

Usage

Standalone (greedy inference)

python

from peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfigimport torch bnb_config = BitsAndBytesConfig(    load_in_4bit=True,    bnb_4bit_compute_dtype=torch.bfloat16,    bnb_4bit_use_double_quant=True,    bnb_4bit_quant_type="nf4",)base = AutoModelForCausalLM.from_pretrained(    "Qwen/Qwen3.5-27B",    quantization_config=bnb_config,    device_map="auto",)model = PeftModel.from_pretrained(base, "Diginyx/Qwen3.5-27B-sft-ep1")tokenizer = AutoTokenizer.from_pretrained("Diginyx/Qwen3.5-27B-sft-ep1")

With PRM-guided inference (best-of-N)

Install Playpen and register the model in model_registry.json:

json

{  "model_name": "Qwen3.5-27B-sft-ep1",  "backend": "huggingface_local",  "huggingface_id": "Qwen/Qwen3.5-27B",  "model_config": {    "premade_chat_template": true,    "load_in_4bit": true,    "chat_template_kwargs": {"enable_thinking": false},    "peft_model": "Diginyx/Qwen3.5-27B-sft-ep1"  }}

Then run:

bash

python examples/trl/prm_eval.py \    --policy-model Qwen3.5-27B-sft-ep1 \    --prm-path Diginyx/Qwen3.5-27B-prm-ep1 \    --game-all \    --n-candidates 4 \    --temperature 0.7 \    --max-tokens 2048

With beam search

bash

python examples/trl/prm_eval.py \    --policy-model Qwen3.5-27B-sft-ep1 \    --prm-path Diginyx/Qwen3.5-27B-prm-ep1 \    --mode beam-search \    --n-candidates 4 \    --num-beam-iterations 20 \    --game-all \    --temperature 0.7 \    --max-tokens 2048

Code

The full training and inference pipeline is available at Diginyx/playpen-prm-code:

Table with columns: Script, Purpose
Script	Purpose
`prm_trainer.py`	PRM rollout collection + training
`prm_eval.py`	Best-of-N and beam search guided inference
`sft_trainer_lora.py`	SFT policy fine-tuning
`eval_validation_one.py`	Plain (no PRM) evaluation
`run_sft.sh`	SLURM job for SFT training
`run_27b_prm_eval.sh`

Companion Models

PRM: Diginyx/Qwen3.5-27B-prm-ep1 — process reward model trained to score turn-level responses

Framework Versions

PEFT 0.19.1
TRL
Transformers
bitsandbytes

Model Details

Developed by: Diginyx
Base model: Qwen/Qwen3.5-27B
Model type: Causal LM — LoRA adapter (PEFT)
Language: English
License: Apache 2.0
Fine-tuning method: QLoRA (4-bit NF4 base + LoRA adapters)
Training framework: TRL + HuggingFace PEFT

Training Methodology

Training pipeline:

Run the base model on all clembench games to collect rollout transcripts
Filter to transcripts with a positive game outcome (win)
Fine-tune on the winning turns using QLoRA

Design decisions:

Positive-only filtering: Using only winning trajectories (rather than all rollouts with reward labels) avoids the model learning from ambiguous partial-credit or losing sequences, keeping the training signal clean.
4-bit QLoRA: Reduces VRAM from ~55 GB to ~14 GB, allowing the full 27B model to train on a single 48 GB A40 alongside optimizer states. This makes training accessible without multi-node tensor parallelism.
LoRA over full fine-tune: Preserves the base model's general language capabilities while adapting the turn-level game response style. The small adapter (r=16) also prevents overfitting on the filtered game corpus.
Max length 1024: Game turns are typically short; truncating at 1024 tokens keeps the full-vocabulary causal-LM logits tensor (batch × seq × 152k vocab) within GPU memory budget.

Training Data

Dataset: colab-potsdam/playpen-data — clembench 2.0 game instances (training split)
Games: All games present in the benchmark (wordle, taboo, reference, clean_up, and others)
Filtering: Turn-level transcripts where the final game outcome was a win
Preprocessing: Chat-templated using Qwen3.5 instruction template with enable_thinking=False

Hyperparameters

Table with columns: Parameter, Value
Parameter	Value
Learning rate	2e-4
LR scheduler	Cosine with warmup
Epochs trained	1 (early stopping on val loss)
Per-device batch size	4
Effective batch size	128 (auto grad-accum across GPUs)
Max sequence length	1024 tokens
LoRA rank (r)	16
LoRA alpha	32
LoRA dropout

Compute

Table with columns: Resource, Details
Resource	Details
Hardware	4× NVIDIA A40 (48 GB)
Cluster	University of Michigan HPC (SLURM)
Training time	~1 epoch over the filtered positive corpus
Total FLOPs	~2.64 × 10¹⁸

Evaluation

Evaluated on the Playpen benchmark (clembench 2.0) using clemscore (quality-weighted success rate across all games) and statscore (static benchmark aggregate).

When used with the companion PRM (Diginyx/Qwen3.5-27B-prm-ep1) in a best-of-N or beam search guided inference setup, this model achieves higher clemscore than the greedy baseline.

Usage

Standalone (greedy inference)

python

from peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfigimport torch bnb_config = BitsAndBytesConfig(    load_in_4bit=True,    bnb_4bit_compute_dtype=torch.bfloat16,    bnb_4bit_use_double_quant=True,    bnb_4bit_quant_type="nf4",)base = AutoModelForCausalLM.from_pretrained(    "Qwen/Qwen3.5-27B",    quantization_config=bnb_config,    device_map="auto",)model = PeftModel.from_pretrained(base, "Diginyx/Qwen3.5-27B-sft-ep1")tokenizer = AutoTokenizer.from_pretrained("Diginyx/Qwen3.5-27B-sft-ep1")

from peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfigimport torch bnb_config = BitsAndBytesConfig(    load_in_4bit=True,    bnb_4bit_compute_dtype=torch.bfloat16,    bnb_4bit_use_double_quant=True,    bnb_4bit_quant_type="nf4",)base = AutoModelForCausalLM.from_pretrained(    "Qwen/Qwen3.5-27B",    quantization_config=bnb_config,    device_map="auto",)model = PeftModel.from_pretrained(base, "Diginyx/Qwen3.5-27B-sft-ep1")tokenizer = AutoTokenizer.from_pretrained("Diginyx/Qwen3.5-27B-sft-ep1")

With PRM-guided inference (best-of-N)

Install Playpen and register the model in model_registry.json:

json

{  "model_name": "Qwen3.5-27B-sft-ep1",  "backend": "huggingface_local",  "huggingface_id": "Qwen/Qwen3.5-27B",  "model_config": {    "premade_chat_template": true,    "load_in_4bit": true,    "chat_template_kwargs": {"enable_thinking": false},    "peft_model": "Diginyx/Qwen3.5-27B-sft-ep1"  }}

{  "model_name": "Qwen3.5-27B-sft-ep1",  "backend": "huggingface_local",  "huggingface_id": "Qwen/Qwen3.5-27B",  "model_config": {    "premade_chat_template": true,    "load_in_4bit": true,    "chat_template_kwargs": {"enable_thinking": false},    "peft_model": "Diginyx/Qwen3.5-27B-sft-ep1"  }}

Then run:

bash

python examples/trl/prm_eval.py \    --policy-model Qwen3.5-27B-sft-ep1 \    --prm-path Diginyx/Qwen3.5-27B-prm-ep1 \    --game-all \    --n-candidates 4 \    --temperature 0.7 \    --max-tokens 2048

python examples/trl/prm_eval.py \    --policy-model Qwen3.5-27B-sft-ep1 \    --prm-path Diginyx/Qwen3.5-27B-prm-ep1 \    --game-all \    --n-candidates 4 \    --temperature 0.7 \    --max-tokens 2048

With beam search

bash

python examples/trl/prm_eval.py \    --policy-model Qwen3.5-27B-sft-ep1 \    --prm-path Diginyx/Qwen3.5-27B-prm-ep1 \    --mode beam-search \    --n-candidates 4 \    --num-beam-iterations 20 \    --game-all \    --temperature 0.7 \    --max-tokens 2048

python examples/trl/prm_eval.py \    --policy-model Qwen3.5-27B-sft-ep1 \    --prm-path Diginyx/Qwen3.5-27B-prm-ep1 \    --mode beam-search \    --n-candidates 4 \    --num-beam-iterations 20 \    --game-all \    --temperature 0.7 \    --max-tokens 2048

Code

The full training and inference pipeline is available at Diginyx/playpen-prm-code:

Table with columns: Script, Purpose
Script	Purpose
`prm_trainer.py`	PRM rollout collection + training
`prm_eval.py`	Best-of-N and beam search guided inference
`sft_trainer_lora.py`	SFT policy fine-tuning
`eval_validation_one.py`	Plain (no PRM) evaluation
`run_sft.sh`	SLURM job for SFT training
`run_27b_prm_eval.sh`

Companion Models

PRM: Diginyx/Qwen3.5-27B-prm-ep1 — process reward model trained to score turn-level responses

Framework Versions

PEFT 0.19.1
TRL
Transformers
bitsandbytes

Qwen3.5-27B-sft-ep1

README

Model Details

Training Methodology

Training Data

Hyperparameters

Compute

Evaluation

Usage

Standalone (greedy inference)

With PRM-guided inference (best-of-N)

With beam search

Code

Companion Models

Framework Versions

Explore FriendliAI today

README

Model Details

Training Methodology

Training Data

Hyperparameters

Compute

Evaluation

Usage

Standalone (greedy inference)

With PRM-guided inference (best-of-N)

With beam search

Code

Companion Models

Framework Versions