Authentic but Not Neutral: Selective Evidence in LLM-Agent Leaderboards
Abstract
AbstractLarge language models are increasingly being used as agents that interact with tools and environments to carry out complex tasks. This development has prompted a growing range of benchmarks and leaderboards for comparing agent performance. Yet the evidence behind these comparisons is not always fully disclosed. Even when every disclosed record is authentic and correctly scored, a reporter with access to the complete ledger can choose a favorable subset. We study whether rank claims remain informative under such adaptive release. An authenticated disclosure frame fixes systems, denominators, bundle bounds, and the canonical claim universe before selection. Worst-case completion keeps every reported edge sound, but a budgeted reporter can still choose which comparisons become certifiable. This separates the soundness of reported comparisons from neutrality in what gets reported. Across four public agent-evaluation ledgers, target-directed release produces positive attention gaps for most eligible targets at intermediate budgets; the effect depends on budget and bundle layout and disappears under an equal-per-system quota in one ledger. We derive an exact polynomial algorithm for the unit-cost target problem and prove weak NP-completeness under heterogeneous costs. These experiments are structural stress tests, not evidence that benchmark maintainers select evidence strategically. They show that certificate soundness does not imply disclosure neutrality.