Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Spectral DPO: Mitigating Validation Collapse via Heavy-Tailed Regularization

AbstractDirect Preference Optimization removes the explicit reward model of RLHF but inherits its central failure mode: the preference-reward proxy can keep improving while downstream reasoning, factuality, or safety degrades. We ask whether this reward overoptimization, which we label validation collapse as an operational name for the trajectory we measure, is visible in weight-space geometry. Drawing on Random Matrix Theory and the Heavy-Tailed Self-Regularisation hypothesis, we propose Spectral DPO, which augments the DPO objective with a stochastic penalty computed from a truncated top-k Lanczos estimate of the spectra of effective adapted projections. The penalty holds the fitted tail exponent near a heavy-tailed reference, caps scale-normalised growth of the largest singular direction, and matches the estimated spectrum to a heavy-tailed target. Across ten matched seeds, three model families, and five benchmarks spanning instruction following, dialogue, mathematical reasoning, factuality, and adversarial robustness, Spectral DPO gains 2.33–2.42 points over the strongest included baseline and sustains accuracy through ten epochs where vanilla DPO degrades. Controls and a post-hoc spectral repair rule out norm clipping, KL anchoring, and length inflation, and support the relevance of spectral geometry without identifying a single causal statistic. All runs use rank-64 LoRA, so the evidence concerns effective-spectrum control under rank-constrained adaptation.

More from EMNLP 2026

All EMNLP 2026 papers →