When Response Variation Does Not Indicate Uncertainty: Hallucination Detection for Multi-Answer Queries
Abstract
AbstractHallucination detection in LLMs often relies on output consistency or model-internal signals. While output-based methods have the advantage of applicability to closed-source models, existing studies often overlook a key limitation: they struggle on tasks that admit multiple valid answers, where variation across generations may reflect legitimate alternative responses rather than hallucinations. In this paper, we propose a question-type-aware hallucination detection framework that distinguishes single-answer and multi-answer regimes. Given a predicted question-type signal, we derive type-conditioned diagnostic features from low- and high-temperature generations, and build two alternative hallucination detectors based on these features: a calibrated type-aware uncertainty estimator and a feature-verbalized few-shot LLM judge. Experiments on NQ-Open, WebQA, and AmbigQA across both open-source and closed-source LLMs demonstrate that our method consistently outperforms conventional semantic clustering approaches and representative baselines in hallucination detection.