Knowing When to Speak: Aligning LLM Honesty via Solvability-Aware Preference Optimization
Abstract
AbstractLarge Language Models often exhibit dishonest behaviors–confidently hallucinating answers to questions beyond their knowledge boundary or hedging on queries they have mastered. Although existing mitigation methods have progressed to establishing the model knowledge boundary, this boundary is still typically treated as a coarse “known vs. unknown” distinction, leaving a finer-grained question underexplored: Does a failure reflect missing knowledge, or merely under-expressed capability? In this paper, we propose KCAPO, a framework that aligns the model’s output behavior with its latent capabilities. It estimates query-level solvability with a consistency-supervised hidden-state probe, then uses its mismatch with response confidence and factual behavior to refine the knowledge boundary into four capability–behavior cases. Based on this fine-grained characterization, we construct preference data to dynamically "Wake Up" dormant knowledge or "Suppress" blind guessing. Furthermore, we introduce an adaptive margin into the DPO objective, which prioritizes correcting responses with severe cognitive dissonance. Extensive experiments on short-form QA and refusal benchmarks show that KCAPO reduces hallucinations without compromising utility and improves selective abstention.