Status
Failed. Composite declined from stage 1 (80.7 → 78.4). Consistency — the target axis — dropped below the SFT baseline (71.0 → 68.8). Every axis regressed from stage 1.
Two root causes: the NLL component of ORPO ran extensive SFT on machine-generated text (250-token completions from a weak 14B model), degrading prose quality through distribution shift. Separately, the preference signal never transferred to generation — training accuracy reached 0.90 but eval composite was flat.
Training Data
Table with columns: Metric, Value| Metric | Value |
|---|
| Pairs | 18,283 |
| Words | 38.9M |
| Selection | Consistency-drift, both sides composite ≥78 |
| Prompt tokens | 1k-4k (3 length buckets) |
| Completion tokens | ~250 each |
- On-policy: same candidate pool as stage 1, finer signal selection
- DeepSeek-V4-Pro judge, consistency axis as primary
- Both sides are competent prose — subtle quality difference
- Secondary signals: originality (+9.1 mean gap), voice (+6.0)
Results
Table with columns: Metric, Value| Metric | Value |
|---|
| NLL loss | 2.52 → 1.09 |
| Accuracy | 0.25 → 0.90 |
| Grad norm | 14 avg (clipped at 20) |
Full epoch completed. Preference metrics climbed slowly, never plateaued. 158/160 test generations clean (2 very short).
logps/rejected flat (both sides good prose). rewards/chosen slowly climbing. Model learned what TO do — constructive, not punitive.
Source data axis contamination
Pairs were selected by consistency gap, but other axes correlated in the source data:
Table with columns: Axis, Mean Gap, Chosen > Rejected| Axis | Mean Gap | Chosen > Rejected |
|---|
| consistency (target) | +21.0 | 100% |
| originality | +9.1 | 88.4% |
| voice | +6.0 | 79.8% |
| garbage | +2.2 | 64.0% |
| repetition | +1.2 | 42.3% |
Chosen was worse on repetition in 57.7% of pairs — consistency selection actively traded off repetition quality.
Eval vs SFT baseline (DeepSeek-V4-Pro judge, 1,024 gens)
Table with columns: Axis, SFT baseline, Post-stage1, Post-stage2| Axis | SFT baseline | Post-stage1 | Post-stage2 |
|---|
| consistency | 71.0 | 72.3 | 68.8 |
| garbage | 83.8 | 86.6 | 84.4 |
| originality | 65.3 | 66.4 | 64.0 |
| repetition | 87.1 | 90.6 | 90.3 |
| voice |
Why it didn't work
- β=0.1 lets NLL dominate. ORPO loss is
NLL(chosen) + β * odds_ratio. At β=0.1 the model mostly does SFT on chosen text, memorizing specific sequences (NLL 2.52→1.09) rather than learning the preference. Training accuracy rises through memorization, not generalization.
- DeepSeek is unreliable on implicit quality axes. Consistency and originality are subjective structural properties that a judge can't reliably measure. The training signal and the eval metric share this limitation.
- ORPO is reference-free. Unlike DPO, there is no KL anchor to the SFT distribution. The policy drifts silently while satisfying chosen > rejected on training data. DPO may be the right algorithm for subtle semantic axes for low-parameter count models.
- Pairs weren't better on all axes. Consistency selection actively selected against repetition (chosen worse 57.7% of the time). The model received contradictory gradients across axes.
Training
- 1×H100 80GB SXM (Unsloth + TRL ORPOTrainer)
- LoRA rank 512, alpha 512 (rsLoRA), continued from stage 1
- 4480 context (measured pair max: 4449 Tekken tokens)
- BF16 base, FP32 AdamW
- ~5.6 hours
Table with columns: Metric, Value| Metric | Value |
|---|
| Steps | 2,286 / 2,286 |
| Learning rate | 1e-5 → 5e-6 (cosine with min LR) |
| Batch size | 8 (1 × grad_accum 8) |
| Beta | 0.1 |
Trained with Unsloth + TRL ORPOTrainer.