Skip to content
conftrace_
2026 EMNLP EMNLP 2026

BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity

AbstractDo language model benchmarks actually measure what practitioners intend them to? Benchmark metadata often obscures how benchmarks operationalize a capability: a "poetry" benchmark may omit haikus, while "instruction-following" benchmarks may test for some arbitrary set of skills. This coarseness prevents practitioners from inspecting how capabilities are operationalized at the item level, leading to unclear estimates of model competence. We introduce BenchBrowser, an auditing infrastructure that retrieves use-case-relevant items from 20+ benchmarks, letting practitioners inspect the evidence behind benchmark claims. Through a human evaluation with ten NLP practitioners, we show that the retrieved items are relevant and help support two meta-evaluations: surfacing content validity threats, where advertised capabilities are unevenly represented across facets, and convergent validity threats, where nominally similar benchmarks induce unstable model rankings. By tying validity concerns to concrete examples, BenchBrowser provides interpretable evidence to calibrate interpretation of benchmark scores and improve future iterations of benchmarks.

More from EMNLP 2026

All EMNLP 2026 papers →