Diginyx

Qwen3.5-27B-prm-ep1

README

License: apache-2.0

Model Details

Developed by: Diginyx
Base model: Qwen/Qwen3.5-27B
Model type: Sequence classifier — LoRA adapter (PEFT), num_labels=1
Output: Scalar logit → sigmoid → P(success | conversation prefix + candidate response)
Language: English
License: Apache 2.0
Fine-tuning method: QLoRA (4-bit NF4 base + LoRA adapters on q_proj, v_proj)
Training framework: TRL + HuggingFace PEFT

Training Methodology

The PRM is trained using soft labels derived from Monte Carlo rollouts (MATH-SHEPHERD style), not human annotations. For each step in a game trajectory, the label is the fraction of independent rollouts from that step that reached a successful game outcome — the expected future success probability.

Rollout pipeline:

Phase 1 — Base trajectory: Play the SFT policy through a full game to collect a base transcript with branch points at each player turn.
Phase 2 — Independent rollouts: From each branch point, run K independent rollouts to game completion, each scored by the game's own clembench BENCH_SCORE metric (0–1 continuous, aborts → 0).
Label construction: The soft label for a step is the mean BENCH_SCORE over all N rollouts from that branch point — the expected bench score reachable from that state.
Training: Fine-tune AutoModelForSequenceClassification (num_labels=1) on (conversation_prefix + candidate_response, soft_label) pairs using MSE/BCE loss.

Design decisions:

Soft labels over hard labels: Using expected bench score (continuous in [0,1]) rather than binary win/loss preserves gradient signal for near-miss trajectories and avoids reward hacking on sparse binary outcomes.
bench scoring strategy: Trained on the game's native clembench bench score rather than a simple win/loss flag, giving the PRM a richer signal that reflects partial progress (e.g., partial credit for partial task completion).
q_proj + v_proj only: Targeting only query and value projections keeps the adapter small and the classification head well-conditioned; full attention-layer targeting OOMed at 4-bit on a 48 GB A40 with sequence classification overhead.
Max length 1024: Conversation prefixes are truncated from the left at 1024 tokens, keeping the most recent game context which carries the most signal for predicting local turn quality.
4-bit QLoRA base: Reduces the base model footprint from ~55 GB to ~14 GB, allowing the classifier head and adapter to fit alongside training on a single A40.

Training Data

Source: Monte Carlo rollouts of Diginyx/Qwen3.5-27B-sft-ep1 on the training split of colab-potsdam/playpen-data (clembench 2.0)
Games: All games in the benchmark suite (game_name="all")
Label type: bench — mean clembench BENCH_SCORE over N rollouts from each branch point
Rollout truncation: Long-game rollouts capped at PRM_MAX_ROLLOUT_ROUNDS rounds past the branch point; partial rollouts labelled with the clembench partial score at cutoff
Branch subsampling: Trajectories with more branch points than the cap are subsampled evenly across the trajectory to bound memory usage

Hyperparameters

Table with columns: Parameter, Value
Parameter	Value
Learning rate	3e-5
LR scheduler	Default (linear)
Max epochs	50 (early stopping, patience=3)
Per-device batch size	4
Gradient accumulation	32
Effective batch size	128
Max sequence length	1024 tokens (truncate left)
LoRA rank (r)	16
LoRA alpha

Compute

Table with columns: Resource, Details
Resource	Details
Hardware	NVIDIA A40 (48 GB) GPUs
Cluster	University of Michigan HPC (SLURM)
Account	chaijy2 / chaijy0

Usage

Score a candidate response

python

import torchimport torch.nn.functional as Ffrom peft import PeftModel, PeftConfigfrom transformers import AutoModelForSequenceClassification, AutoTokenizer, BitsAndBytesConfig prm_path = "Diginyx/Qwen3.5-27B-prm-ep1"peft_cfg = PeftConfig.from_pretrained(prm_path) bnb_config = BitsAndBytesConfig(    load_in_4bit=True,    bnb_4bit_compute_dtype=torch.bfloat16,    bnb_4bit_use_double_quant=True,    bnb_4bit_quant_type="nf4",)base = AutoModelForSequenceClassification.from_pretrained(    peft_cfg.base_model_name_or_path,    num_labels=1,    quantization_config=bnb_config,    device_map="auto",)prm = PeftModel.from_pretrained(base, prm_path)tokenizer = AutoTokenizer.from_pretrained(prm_path)prm.eval() # messages: the conversation history up to the current turn# candidate: the response to scoremessages = [    {"role": "user", "content": "Guess a 5-letter word."},    {"role": "assistant", "content": "Guess: crane"},]text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=1024,                   truncation_side="left").to("cuda")with torch.no_grad():    score = torch.sigmoid(prm(**inputs).logits[0, 0]).item()print(f"P(success) = {score:.4f}")

Best-of-N inference with the policy

Install Playpen and run:

bash

python examples/trl/prm_eval.py \    --policy-model Qwen3.5-27B-sft-ep1 \    --prm-path Diginyx/Qwen3.5-27B-prm-ep1 \    --game-all \    --n-candidates 4 \    --temperature 0.7 \    --max-tokens 2048

Beam search inference

bash

python examples/trl/prm_eval.py \    --policy-model Qwen3.5-27B-sft-ep1 \    --prm-path Diginyx/Qwen3.5-27B-prm-ep1 \    --mode beam-search \    --n-candidates 4 \    --num-beam-iterations 20 \    --game-all \    --temperature 0.7 \    --max-tokens 2048

Code

The full training and inference pipeline — including rollout collection, PRM training, SFT training, best-of-N, and beam search — is available at Diginyx/playpen-prm-code:

Table with columns: Script, Purpose
Script	Purpose
`prm_trainer.py`	PRM rollout collection + training
`prm_train_from_records.py`	Train PRM from pre-collected rollout records
`prm_eval.py`	Best-of-N and beam search guided inference
`sft_trainer_lora.py`	SFT policy fine-tuning
`eval_validation_one.py`	Plain (no PRM) evaluation

Companion Models

Policy: Diginyx/Qwen3.5-27B-sft-ep1 — SFT fine-tuned policy model

Framework Versions

PEFT 0.19.1
TRL
Transformers
bitsandbytes

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Diginyx

Model Tree

Base

Qwen/Qwen3.5-27B

Adapter

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Details

Developed by: Diginyx
Base model: Qwen/Qwen3.5-27B
Model type: Sequence classifier — LoRA adapter (PEFT), num_labels=1
Output: Scalar logit → sigmoid → P(success | conversation prefix + candidate response)
Language: English
License: Apache 2.0
Fine-tuning method: QLoRA (4-bit NF4 base + LoRA adapters on q_proj, v_proj)
Training framework: TRL + HuggingFace PEFT

Training Methodology

Rollout pipeline:

Phase 1 — Base trajectory: Play the SFT policy through a full game to collect a base transcript with branch points at each player turn.
Phase 2 — Independent rollouts: From each branch point, run K independent rollouts to game completion, each scored by the game's own clembench BENCH_SCORE metric (0–1 continuous, aborts → 0).
Label construction: The soft label for a step is the mean BENCH_SCORE over all N rollouts from that branch point — the expected bench score reachable from that state.
Training: Fine-tune AutoModelForSequenceClassification (num_labels=1) on (conversation_prefix + candidate_response, soft_label) pairs using MSE/BCE loss.

Design decisions:

Soft labels over hard labels: Using expected bench score (continuous in [0,1]) rather than binary win/loss preserves gradient signal for near-miss trajectories and avoids reward hacking on sparse binary outcomes.
bench scoring strategy: Trained on the game's native clembench bench score rather than a simple win/loss flag, giving the PRM a richer signal that reflects partial progress (e.g., partial credit for partial task completion).
q_proj + v_proj only: Targeting only query and value projections keeps the adapter small and the classification head well-conditioned; full attention-layer targeting OOMed at 4-bit on a 48 GB A40 with sequence classification overhead.
Max length 1024: Conversation prefixes are truncated from the left at 1024 tokens, keeping the most recent game context which carries the most signal for predicting local turn quality.
4-bit QLoRA base: Reduces the base model footprint from ~55 GB to ~14 GB, allowing the classifier head and adapter to fit alongside training on a single A40.

Training Data

Source: Monte Carlo rollouts of Diginyx/Qwen3.5-27B-sft-ep1 on the training split of colab-potsdam/playpen-data (clembench 2.0)
Games: All games in the benchmark suite (game_name="all")
Label type: bench — mean clembench BENCH_SCORE over N rollouts from each branch point
Rollout truncation: Long-game rollouts capped at PRM_MAX_ROLLOUT_ROUNDS rounds past the branch point; partial rollouts labelled with the clembench partial score at cutoff
Branch subsampling: Trajectories with more branch points than the cap are subsampled evenly across the trajectory to bound memory usage

Hyperparameters

Table with columns: Parameter, Value
Parameter	Value
Learning rate	3e-5
LR scheduler	Default (linear)
Max epochs	50 (early stopping, patience=3)
Per-device batch size	4
Gradient accumulation	32
Effective batch size	128
Max sequence length	1024 tokens (truncate left)
LoRA rank (r)	16
LoRA alpha

Compute

Table with columns: Resource, Details
Resource	Details
Hardware	NVIDIA A40 (48 GB) GPUs
Cluster	University of Michigan HPC (SLURM)
Account	chaijy2 / chaijy0

Usage

Score a candidate response

python

import torchimport torch.nn.functional as Ffrom peft import PeftModel, PeftConfigfrom transformers import AutoModelForSequenceClassification, AutoTokenizer, BitsAndBytesConfig prm_path = "Diginyx/Qwen3.5-27B-prm-ep1"peft_cfg = PeftConfig.from_pretrained(prm_path) bnb_config = BitsAndBytesConfig(    load_in_4bit=True,    bnb_4bit_compute_dtype=torch.bfloat16,    bnb_4bit_use_double_quant=True,    bnb_4bit_quant_type="nf4",)base = AutoModelForSequenceClassification.from_pretrained(    peft_cfg.base_model_name_or_path,    num_labels=1,    quantization_config=bnb_config,    device_map="auto",)prm = PeftModel.from_pretrained(base, prm_path)tokenizer = AutoTokenizer.from_pretrained(prm_path)prm.eval() # messages: the conversation history up to the current turn# candidate: the response to scoremessages = [    {"role": "user", "content": "Guess a 5-letter word."},    {"role": "assistant", "content": "Guess: crane"},]text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=1024,                   truncation_side="left").to("cuda")with torch.no_grad():    score = torch.sigmoid(prm(**inputs).logits[0, 0]).item()print(f"P(success) = {score:.4f}")

import torchimport torch.nn.functional as Ffrom peft import PeftModel, PeftConfigfrom transformers import AutoModelForSequenceClassification, AutoTokenizer, BitsAndBytesConfig prm_path = "Diginyx/Qwen3.5-27B-prm-ep1"peft_cfg = PeftConfig.from_pretrained(prm_path) bnb_config = BitsAndBytesConfig(    load_in_4bit=True,    bnb_4bit_compute_dtype=torch.bfloat16,    bnb_4bit_use_double_quant=True,    bnb_4bit_quant_type="nf4",)base = AutoModelForSequenceClassification.from_pretrained(    peft_cfg.base_model_name_or_path,    num_labels=1,    quantization_config=bnb_config,    device_map="auto",)prm = PeftModel.from_pretrained(base, prm_path)tokenizer = AutoTokenizer.from_pretrained(prm_path)prm.eval() # messages: the conversation history up to the current turn# candidate: the response to scoremessages = [    {"role": "user", "content": "Guess a 5-letter word."},    {"role": "assistant", "content": "Guess: crane"},]text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=1024,                   truncation_side="left").to("cuda")with torch.no_grad():    score = torch.sigmoid(prm(**inputs).logits[0, 0]).item()print(f"P(success) = {score:.4f}")

Best-of-N inference with the policy

Install Playpen and run:

bash

python examples/trl/prm_eval.py \    --policy-model Qwen3.5-27B-sft-ep1 \    --prm-path Diginyx/Qwen3.5-27B-prm-ep1 \    --game-all \    --n-candidates 4 \    --temperature 0.7 \    --max-tokens 2048

python examples/trl/prm_eval.py \    --policy-model Qwen3.5-27B-sft-ep1 \    --prm-path Diginyx/Qwen3.5-27B-prm-ep1 \    --game-all \    --n-candidates 4 \    --temperature 0.7 \    --max-tokens 2048

Beam search inference

bash

python examples/trl/prm_eval.py \    --policy-model Qwen3.5-27B-sft-ep1 \    --prm-path Diginyx/Qwen3.5-27B-prm-ep1 \    --mode beam-search \    --n-candidates 4 \    --num-beam-iterations 20 \    --game-all \    --temperature 0.7 \    --max-tokens 2048

python examples/trl/prm_eval.py \    --policy-model Qwen3.5-27B-sft-ep1 \    --prm-path Diginyx/Qwen3.5-27B-prm-ep1 \    --mode beam-search \    --n-candidates 4 \    --num-beam-iterations 20 \    --game-all \    --temperature 0.7 \    --max-tokens 2048

Code

The full training and inference pipeline — including rollout collection, PRM training, SFT training, best-of-N, and beam search — is available at Diginyx/playpen-prm-code:

Table with columns: Script, Purpose
Script	Purpose
`prm_trainer.py`	PRM rollout collection + training
`prm_train_from_records.py`	Train PRM from pre-collected rollout records
`prm_eval.py`	Best-of-N and beam search guided inference
`sft_trainer_lora.py`	SFT policy fine-tuning
`eval_validation_one.py`	Plain (no PRM) evaluation

Companion Models

Policy: Diginyx/Qwen3.5-27B-sft-ep1 — SFT fine-tuned policy model

Framework Versions

PEFT 0.19.1
TRL
Transformers
bitsandbytes