TL;DR

• Behavior ROUGE-L 0.911 — best of all variants (was 0.036 in the natural-distribution baseline at lr=2e-4).
• Perception ROUGE-L 0.615 — best of all variants.
• Prediction ROUGE-L 0.368 — worst of all post-LoRA variants (vs 0.659 for natural-distribution at lr=2e-4). Cost of forcing uniform answer-pattern proportions.
• Overall ROUGE-L 0.518 — a regression vs the simpler lr=1e-4 fix.

Eval results (3,770-sample DriveLM front-arc, vLLM)

Per question category (ROUGE-L)

Why the prediction regression

DriveLM's prediction-category eval set has a natural distribution of ~2/38/16/44% Yes/No/None-pattern/other. The natural-1024 training data matched that distribution almost exactly (7/119/49/136). Uniform stratification forced 50/50/50/100 — the model never learned that "No is 3× more likely than Yes" for prediction, and at inference it gives wrong base rates. The 0.29 ROUGE-L drop on a 1,181-sample category is the cost.

The proportional sibling addresses this by preserving natural within-category proportions with a min-floor on rare classes.

Training Details

Identical to the natural-1024 sibling except for the sample composition:

The lesson

Failure mode analysis pointed at data, but the actual fix was a hyperparameter. The behavior collapse we attributed to "only 10 behavior samples in training" was mostly the lr=2e-4 being slightly too aggressive — with lr=1e-4, those 10 samples are enough to keep behavior at 0.877. The data-side stratification fixed behavior even harder but at the cost of prediction. Net: the LR fix is the better lever.

Limitations

Same as the series.

License

Apache-2.0.