Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Auditing LLM Benchmarks with Item Response Theory

AbstractLLM benchmark labels are frozen at release and silently copied into downstream benchmarks, errors and all. Our Item Response Theory indicator surfaces these mislabels at 95% precision in the top 200, across 114 models on seven preference and multiple-choice benchmarks, outperforming a supervised classifier. We trace these errors to mechanical labeling rules, upstream errors inherited unchanged from source datasets, and ambiguous items with no defensible single key. The same fit reveals that reward models specialize in stylistic preference over factual knowledge, and exposes one frontier reward model that agrees with detected mislabels at 78% accuracy versus 38% for its peers, consistent with contamination or benchmark-specific over-optimization.

More from EMNLP 2026

All EMNLP 2026 papers →