GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
cbyabush
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
cbyabush
Model Tree
openai-community/gpt2
Input Modalities
Output Modalities
Supported Functionality
Direct Preference Optimization is a modern alignment technique that teaches a language model to prefer certain responses over others using human preference data. Unlike standard RLHF — which requires a separate reward model and reinforcement learning — DPO collapses both into a single supervised loss:
markdown
L_DPO = -log σ( β * [(log π_θ(yw|x) - log π_θ(yl|x)) - (log π_ref(yw|x) - log π_ref(yl|x))] )
where yw is the chosen response, yl is the rejected response, π_θ is the policy being trained, and π_ref is a frozen reference model.
python
from transformers import AutoModelForCausalLM, AutoTokenizermodel = AutoModelForCausalLM.from_pretrained("cbyabush/gpt2-dpo-ultrafeedback")tokenizer = AutoTokenizer.from_pretrained("cbyabush/gpt2-dpo-ultrafeedback")prompt = "User: What is the capital of France?\nAssistant: "inputs = tokenizer(prompt, return_tensors="pt")outputs = model.generate(**inputs, max_new_tokens=64, do_sample=False)print(tokenizer.decode(outputs[0], skip_special_tokens=True))
A subset of UltraFeedback, a large-scale preference dataset of (prompt, chosen, rejected) triples covering diverse instruction-following tasks.
The policy model starts from GPT-2 weights. A frozen copy serves as the reference model throughout training. The DPO loss encourages the policy to assign higher relative probability to chosen responses compared to rejected ones, anchored by the reference.
| Parameter | Value |
|---|---|
| Base model | gpt2 |
| β (beta) | 0.1 |
| Max prompt tokens | 96 |
| Max response tokens | 64 |
| Max total length | 160 |
| Batch size | 4 |
| Gradient accumulation steps | 4 |
| Effective batch size | 16 |
| Learning rate | 1e-5 |
Every component was implemented from scratch in pure PyTorch:
Preference accuracy measures how often the policy assigns higher log-probability to the chosen response than the rejected response, relative to the reference model:
markdown
accuracy = mean[ (log π_θ(yw|x) - log π_θ(yl|x)) > (log π_ref(yw|x) - log π_ref(yl|x)) ]
bibtex
@article{rafailov2023direct,title={Direct Preference Optimization: Your Language Model is Secretly a Reward Model},author={Rafailov, Rafael and Sharma, Archit and Mitchell, Eric and Ermon, Stefano and Manning, Christopher D and Finn, Chelsea},journal={arXiv preprint arXiv:2305.18290},year={2023}}@article{cui2023ultrafeedback,title={UltraFeedback: Boosting Language Models with High-quality Feedback},author={Cui, Ganqu and Yuan, Lifan and Ding, Ning and Yao, Guanming and Zhu, Wei and Ni, Yuan and Xie, Guotong and Liu, Zhiyuan and Sun, Maosong},journal={arXiv preprint arXiv:2310.01377},year={2023}}
| Optimizer | AdamW |
| Gradient clipping | 1.0 |
| Epochs | 1 |
| Seed | 13 |