sidmhuggingface

rag-dpo-qwen2.5-0.5b

README

License: apache-2.0

Training

Method: DPO (trl DPOTrainer), beta=0.1, 3 epochs, LoRA r=16 continuing from the SFT adapter
Data: 40 preference pairs (34 train / 6 eval), questions distinct from the SFT training set
Training-time signal: reward accuracy 0.79 -> 0.94 -> 1.0 across 3 epochs, reward margin 0.37 -> 1.23, eval loss 0.39 -> 0.32 - DPO clearly learned to separate chosen from rejected during training.

Honest evaluation: DPO did not clearly beat SFT on held-out generation quality

Pairwise win rate on 20 held-out questions, judged with gemma3-27b-it - deliberately a different model family from the qwen3-235b DPO teacher, to avoid the judge favoring outputs that resemble its own family. Each pair was judged twice with (SFT, DPO) presentation order swapped and reconciled to a winner or "tie" - a standard control for LLM-judge position bias.

Table with columns: Win rate
	Win rate
SFT wins	35%
DPO wins	20%
Tie	45%

SFT alone won more often than the DPO-aligned model. This is reported as-is, not tuned away: an earlier check with a different judge (before the bias controls above were added, and before that run hit an unrelated API error) showed the same mixed pattern, which cross-checks that this isn't a one-off artifact of the specific judge model.

The likely explanation is that the training signal here is thin by construction: a 0.5B parameter model, LoRA rank 16, and only 40 preference pairs. The gap is between "the model learned to prefer chosen over rejected during training" (which the reward-accuracy curve above shows clearly) and "that generalizes to better held-out answers judged independently" (which it does not, at this scale). A larger preference set or more epochs might change this - untested here, and worth flagging as a natural next step rather than retroactively "fixing" the number by re-running until it looks better.

Sample generations (held-out questions)

Q: What is the significance of treewidth in determining the efficiency of exact inference in graphical models?

SFT: "Treewidth plays a crucial role in evaluating the complexity of exact inference for graphical models. A model with low treewidth can be more efficiently solved using approximate algorithms..."
DPO: "Treewidth plays a crucial role in assessing the complexity of exact inference for graphical models. A model with low treewidth (e.g., 2 or less) generally has simpler structures..."
Judge verdict: SFT

Q: What method was used to generate in-distribution adversarial attacks in the study on CNN robustness for lumbar disk shape reconstruction?

SFT: "The study generated in-distribution adversarial attacks using a combination of adversarial training and data augmentation techniques."
DPO: "The study utilized a combination of adversarial training and in-distribution adversarial attacks to improve the robustness of CNNs for lumbar disk shape reconstruction."
Judge verdict: tie

Usage

python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
model = PeftModel.from_pretrained(base, "sidmhuggingface/rag-dpo-qwen2.5-0.5b")
tokenizer = AutoTokenizer.from_pretrained("sidmhuggingface/rag-dpo-qwen2.5-0.5b")

Framework versions

PEFT 0.19.1
TRL 0.24.0
Transformers 5.12.1
PyTorch 2.11.0

Citation

bibtex
@inproceedings{rafailov2023direct,
    title        = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},
    author       = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn},
    year         = 2023,
    booktitle    = {Advances in Neural Information Processing Systems 36},
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

sidmhuggingface

Model Tree

Base

Qwen/Qwen2.5-0.5B-Instruct

Adapter

this model

Input Modalities

Text

Output Modalities