Pipeline

markdown
Copy code
Qwen/Qwen3-0.6B (base)
        ↓ SFT with LoRA
qwen3-0.6b-sft-dolly 
        ↓ DPO
qwen3-0.6b-dpo-ultrafeedback (this model)

DPO Configuration

Dataset

• Name: HuggingFaceH4/ultrafeedback_binarized
• Split: train_prefs
• Subset: 2,000 samples (seed=42)
• Format: prompt / chosen / rejected pairs

Results

Note on Metrics

DPO scores lower than SFT on BLEU/BERTScore. This is expected — DPO optimizes for preference alignment, not surface-level n-gram overlap. The model produces more natural, measured responses.

How to Use

python
Copy code
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

tokenizer = AutoTokenizer.from_pretrained(
    "Shaheer05/qwen3-0.6b-dpo-ultrafeedback",
    trust_remote_code=True
)

# Load SFT first, then DPO on top
base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-0.6B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)
sft_model = PeftModel.from_pretrained(
    base,
    "Shaheer05/qwen3-0.6b-sft-dolly"
)
sft_model = sft_model.merge_and_unload()

model = PeftModel.from_pretrained(
    sft_model,
    "Shaheer05/qwen3-0.6b-dpo-ultrafeedback"
)

# Inference
messages = [{"role": "user", "content": "What is machine learning?"}]
text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Training Platform

• Google Colab (Free Tier) — NVIDIA T4 GPU