Skip to content
conftrace_
2026 EMNLP EMNLP 2026

The Benchmark Is Not the Construct: Toward Distribution-Based Evaluation in NLP

AbstractCurrent NLP evaluation often treats a scalar score on a fixed benchmark dataset as if it directly measured the construct the benchmark is intended to represent. This Measurement-as-Instance paradigm is appropriate for a fixed evaluation configuration, but can suppress uncertainty, variance, and robustness when a single result is used to support broader capability or generalisation claims. In this position paper, we argue for extending static evaluation with Measurement-as-Distribution: a structured evaluation procedure involving principled data generation, validated perturbations, repeated evaluation, and distributional reporting. The framework is configurable: the amount of distributional evidence required should scale with the strength and breadth of the claims being made. We synthesise existing robustness, dynamic-evaluation, prompt-sensitivity, and uncertainty-reporting practices into a general measurement-theoretic framework for distribution-based NLP evaluation and provide a call to action for more rigorous reporting. Using the protocol 𝒫 = (𝒞, G, 𝛷, E, R), we identify three risks of relying exclusively on fixed scalar evaluation and derive implications for peer review, benchmark publication, leaderboard reporting, and computational cost. Two case studies demonstrate feasibility: a SQuAD evaluation of 14 models reveals modest but axis-specific performance changes and stable overall rankings with uncertainty-aware comparisons, while a verifier-backed Knights-and-Knaves study illustrates how perturbations of rigid logical tasks can be generated and validated formally.

More from EMNLP 2026

All EMNLP 2026 papers →