Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Sparse-Positive Hallucination Evaluation Entangles Detection and Prevalence

AbstractSpan-level hallucination metrics aggregate outputs whose references contain hallucinated spans with outputs whose references are empty. Under scoring rules that award a perfect match to an empty prediction against empty gold, the aggregate entangles detection performance with empty-gold prevalence. We write the exact decomposition as s=ea+(1-e)d, where e is empty-gold prevalence, a is performance on empty-gold items, and d is performance on non-empty-gold items. Under the benchmark’s metric conventions, an empty submission identifies e from leaderboard values without access to closed labels. Applied to a multilingual vision-language hallucination benchmark, this recovers rates from 0.1549 to 0.2554. Assuming perfect empty-gold performance gives a maximum contribution equal to 47.3–64.8% of the mean team score and a conservative lower bound on d. Under this bound, the highest-ranked language moves from best for 17 of 18 teams to best for none and falls from rank 1 to rank 3. Released annotations show that union-over-annotators gold retains 4.09 times as many characters as majority-agreement gold, while 71.7% of annotated spans were marked by one of three annotators. Cross-split comparisons therefore require reporting both empty-gold prevalence and system performance on empty-gold items alongside aggregate scores.

More from EMNLP 2026

All EMNLP 2026 papers →