Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Stabilizing Hard-Negative Preference Optimization for Medical LLMs

AbstractHard-negative preference optimization should improve support for ground-truth preferred responses while limiting drift from the reference model. In Japanese medical instruction tuning, we find that standard DPO and IPO improve the preferred–dispreferred gap but can leave preferred-response likelihood below the reference model and induce growing sequence-level KL divergence. Motivated by this failure-mode analysis, we propose a stabilized IPO objective that combines preferred-response anchoring with explicit KL control. Ablations show that positive anchoring improves preferred-response support, while KL control suppresses reference-model drift; their combination preserves anchoring gains while further reducing drift. These results suggest that reliable medical preference optimization benefits from both preferred-response anchoring and drift control.

More from EMNLP 2026

All EMNLP 2026 papers →