2026
ICML
ICML 2026
When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
Abstract
Artificial intelligence benchmarks are an important mechanism to measure model progress and guide deployment decisions. However, benchmarks quickly “saturate”, making it difficult to differentiate models and diminishing their long-term value. In this study, we define benchmark saturation and analyze it across 60 language model benchmarks using 14 properties that relate to saturation. We find that nearly half of our benchmarks exhibit saturation, with rates increasing with age. Further, we find that resilience to saturation is impacted by expert-curation, not by public test data. Our results suggest that design choices can extend benchmark longevity and inform more durable evaluation approaches.
Authors
Mubashara Akhtar,
Anka Reuel,
Prajna Soni,
Sanchit Ahuja,
Pawan Sasanka Ammanamanchi,
Ruchit Rawal,
Vilém Zouhar,
Srishti Yadav,
Chenxi Whitehouse,
Dayeon Ki,
Jennifer Mickel,
Leshem Choshen,
Marek Suppa,
Jan Batzner,
Jenny Chim,
Jeba Sania,
Yanan Long,
Hossein A. Rahmani,
Christina Q Knight,
Yiyang Nan,
Jyoutir Raj,
Yu Fan,
Shubham Singh,
Subramanyam Sahoo,
Eliya Habba,
Usman Gohar,
Siddhesh Milind Pawar,
Robert Scholz,
Arjun Subramonian,
Jingwei Ni,
Mykel Kochenderfer,
Sanmi Koyejo,
Mrinmaya Sachan,
Stella Biderman,
Zeerak Talat,
Avijit Ghosh,
Irene Solaiman