shubhamrgandhi

Qwen3-4B-DPO-prm-pairs

README

License: apache-2.0

Results — read before using

The run improved over its own baseline but did not exceed chance on pairwise preference accuracy. Reported honestly:

Table with columns: epoch, eval_loss, eval_rewards/accuracies, eval_rewards/margins
epoch	eval_loss	eval_rewards/accuracies	eval_rewards/margins
0.00 (baseline)	0.6914	0.0000 *	0.0000
0.23	0.7172	0.3187	−0.0073
0.46	0.6926	0.3875	+0.0746
0.69	0.6925	0.4562	+0.0866
0.92	0.6949	0.4313	+0.0877
1.00 (final)	0.6680	0.4375	+0.1310

* At step 0 the policy is the reference model, so both rewards are exactly 0 and the strict chosen > rejected comparison is False for every pair. That 0 is a degenerate tie, not a meaningful score.

Final train_loss 0.666.

Interpretation. Eval loss ended below baseline (0.668 vs 0.6914) and the reward margin grew steadily to +0.131, so the model does separate some pairs. But rewards/accuracies finished at 0.4375, below the 0.5 chance line — a positive mean margin combined with a sub-chance win rate means the average is carried by a minority of strongly-separated pairs while most pairs remain slightly mis-ranked.

The metric curve was still climbing at the end of epoch 1, so this looks under-trained rather than converged: only 44 optimizer steps were taken at learning_rate=5e-7. Treat this as a checkpoint of record, not a model that reliably beats its base at preference ranking.

Training details

Base model: Qwen/Qwen3-4B-Instruct-2507 (4.02B params)
Method: DPO (trl.DPOTrainer), β = 0.1 (default)
Data: 1,541 preference pairs (ShareGPT-style messages + chosen/rejected), split 90/10 → 1,386 train / 155 eval (seed 42). Pairs are pre-filtered to fit whole within 8,192 tokens; median length ≈ 7.7k tokens.
max_length: 8192 (nothing truncated)
Precision: bf16 (mixed precision via Accelerate)
Hardware: 8 × NVIDIA L40S (46 GB), single node
Parallelism: DeepSpeed ZeRO-3, no offload, no gradient checkpointing (peak ≈ 41.5 GB/GPU)
Batching: per_device_train_batch_size=1 × gradient_accumulation_steps=4 × 8 GPUs = → 44 optimizer steps for 1 epoch

Reproducing

bash
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 accelerate launch \
    --config_file examples/accelerate_configs/deepspeed_zero3.yaml \
    dpo_prm_pairs.py

Known limitations

Sub-chance pairwise preference accuracy (see above) — do not assume it out-ranks the base model on this task.
Under-trained at 44 optimizer steps; a larger step count (e.g. halving gradient_accumulation_steps to restore ~87 steps) or a higher learning rate is the obvious next experiment.
Single epoch, single seed, no hyperparameter search.
Evaluated only on a 155-pair in-distribution holdout. No general-capability benchmarks were run, so regression on base-model abilities is unmeasured.
Trained on long-context (≈7.7k token) agent traces; behavior on short prompts is untested.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

shubhamrgandhi

Model Tree

Base

Qwen/Qwen3-4B-Instruct-2507

Fine-tuned

this model

Input Modalities

Text

Output Modalities