Skip to content
conftrace_
2026 ICML ICML 2026

Capability Traps in DPO

Direct Preference Optimization (DPO) is often tuned by selecting checkpoints with high preference margins. We show that this selection rule can fail. Across dense $\beta$ sweeps in three 7B open-weight families under controlled DPO recipes, the DPO preference margin can strongly anticorrelate with capability probes, with the strongest observed case in Llama logic probes (Pearson $r=-0.91$, $n=13$). In Mistral, matched-duration controls show that transient exposure to higher $\beta$ produces persistent arithmetic ($d_z = 3.73$, $p = 0.002$) and format ($d_z = 3.38$, $p = 0.003$) degradation relative to a duration-matched constant-$\beta$ run, while logic and sycophancy show no significant effect. An expanded 14-probe logic suite further shows that aggregate capability scores can hide stable opposition between probe clusters: direct-inference probes are consistently negative while fallacy-detection probes are consistently positive, with 13 of 14 probes sign-stable across 3 seeds and 3 $\beta$ values. Sensitivity profiles differ across architectures: Mistral shows elevated seed variance near $\beta \approx 10^{-2}$, Llama shows capability-specific rigidity, and Qwen trades off smoothly. These findings motivate probe-resolved $\beta$ sweeps rather than margin-based checkpoint selection.

More from ICML 2026

All ICML 2026 papers →