GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
narinzar
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
narinzar
Model Tree
openai-community/gpt2
Input Modalities
Output Modalities
Supported Functionality
gpt2 (124M).src/dpo.py), policy trained
against a frozen reference copy of gpt2.src/reward.py),
not by humans. Candidates are sampled from the base gpt2 itself.please.This upload is the full fine-tuned model (safetensors + tokenizer), loadable
directly with transformers.
Measured on a single RTX 5090 (24 GB). 286 train / 72 eval pairs, beta=0.1,
lr=1e-5, batch size 8, 3 epochs.
| Metric | Value |
|---|---|
| Win-rate before DPO (policy vs ref) | 0.500 |
| Win-rate after DPO (policy vs ref) | 0.812 |
| Final DPO loss | 0.199 |
Win-rate is judged by the same reward heuristic that labeled the pairs, so the dataset and the metric are aligned. Before training the policy is a copy of the reference, so win-rate sits at 0.500; after DPO it rises to 0.812. This is a small-scale demo, so the effect size is bounded by the tiny model and dataset; the point is the mechanism and the before/after direction, not a headline score.
python
from transformers import AutoModelForCausalLM, AutoTokenizertok = AutoTokenizer.from_pretrained("narinzar/dpo-finetune-demo")model = AutoModelForCausalLM.from_pretrained("narinzar/dpo-finetune-demo")prompt = "How do I reset my password?"ids = tok(prompt, return_tensors="pt")out = model.generate(**ids, max_new_tokens=48, pad_token_id=tok.eos_token_id)print(tok.decode(out[0], skip_special_tokens=True))
gpt2's limitations and biases.MIT.