hw4-f26-dpo-ref API & Inference Endpoint

Same data and settings as assignment runs:

Software: TRL 1.16.0 DPOTrainer, sigmoid loss, β = 0.1, no label smoothing; reference log-probabilities precomputed.
Data: 2,000 preference pairs from HuggingFaceH4/ultrafeedback_binarized (train_prefs), keeping only pairs where both answers are ≤ 256 tokens.
Optimization: 1 epoch, batch size 8 (250 steps); learning rate 1e-5, constant, no warmup; gradient-norm clip 1.0; bf16; seed 0.
LoRA: rank 8, α 16, no dropout, on q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj.
Compute: about 8 minutes of training on one A100.

Full settings: reference_config.json.

Evaluated with the HW4 evaluation code (hw4_lib.llm.evaluate_adapter):

Table with columns: Metric, Value
Metric	Value
Held-out pair accuracy (all 816 filtered `test_prefs` pairs)	0.652
KL from the base model (100 held-out prompts)	0.12
Mean answer length/share hitting the 256-token cap	115 tokens/19%
Training reward model `Skywork/Skywork-Reward-V2-Qwen3-0.6B` (mean score)	−1.06
Held-out reward model `Skywork/Skywork-Reward-V2-Llama-3.2-1B` (mean score)	2.06

Reward-model scores have an arbitrary offset, so compare them only as changes from the base model on the same prompts (reward-vs-KL plane does this).

Table with columns: File, Contents
File	Contents
`adapter_model.safetensors`, `adapter_config.json`	the LoRA adapter (bf16)
`metrics.jsonl`	per-step training metrics (loss, margin, accuracy, chosen/rejected log-probs)
`trl_log_history.json`	TRL's raw training log
`evals.jsonl`, `plane/final.jsonl`	end-of-run evaluation summary and per-prompt scores on the plane prompts
`samples.jsonl`	greedy answers to the plane prompts
`pairs_final.jsonl`	per-pair results of the held-out pair accuracy
`reference_config.json`	the exact HW4 and TRL settings

Same data and settings as assignment runs:

Software: TRL 1.16.0 DPOTrainer, sigmoid loss, β = 0.1, no label smoothing; reference log-probabilities precomputed.
Data: 2,000 preference pairs from HuggingFaceH4/ultrafeedback_binarized (train_prefs), keeping only pairs where both answers are ≤ 256 tokens.
Optimization: 1 epoch, batch size 8 (250 steps); learning rate 1e-5, constant, no warmup; gradient-norm clip 1.0; bf16; seed 0.
LoRA: rank 8, α 16, no dropout, on q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj.
Compute: about 8 minutes of training on one A100.

Full settings: reference_config.json.

Evaluated with the HW4 evaluation code (hw4_lib.llm.evaluate_adapter):

Table with columns: Metric, Value
Metric	Value
Held-out pair accuracy (all 816 filtered `test_prefs` pairs)	0.652
KL from the base model (100 held-out prompts)	0.12
Mean answer length/share hitting the 256-token cap	115 tokens/19%
Training reward model `Skywork/Skywork-Reward-V2-Qwen3-0.6B` (mean score)	−1.06
Held-out reward model `Skywork/Skywork-Reward-V2-Llama-3.2-1B` (mean score)	2.06

Reward-model scores have an arbitrary offset, so compare them only as changes from the base model on the same prompts (reward-vs-KL plane does this).

Table with columns: File, Contents
File	Contents
`adapter_model.safetensors`, `adapter_config.json`	the LoRA adapter (bf16)
`metrics.jsonl`	per-step training metrics (loss, margin, accuracy, chosen/rejected log-probs)
`trl_log_history.json`	TRL's raw training log
`evals.jsonl`, `plane/final.jsonl`	end-of-run evaluation summary and per-prompt scores on the plane prompts
`samples.jsonl`	greedy answers to the plane prompts
`pairs_final.jsonl`	per-pair results of the held-out pair accuracy
`reference_config.json`	the exact HW4 and TRL settings

hw4-f26-dpo-ref