Do Benchmarks Measure the Same Thing? Evaluation-Free Benchmark Redundancy Analysis with SAE
Abstract
AbstractThe rapid growth of LLM evaluation resources makes exhaustive benchmark curation increasingly difficult. Existing efficient benchmarking methods reduce evaluation cost by selecting representative subsets or predicting full-benchmark performance, but typically require evaluation results from many models and additional training. In this paper, we ask a more fundamental question: to what extent do benchmarks actually measure different things? We study this question at two levels: intra-benchmark redundancy, which supports benchmark compression, and inter-benchmark similarity, which supports evaluation-suite design. We first formalize both notions through a functionalist behavioral criterion, and then propose an evaluation-free proxy based on Sparse Autoencoder (SAE) features for redundancy diagnosis. Our central hypothesis is that behavioral overlap can be approximated by feature overlap. This hypothesis yields level-specific metrics based on feature-coverage saturation within benchmarks and feature-footprint overlap across benchmarks. Experiments show that SAE feature-based redundancy strongly correlates with performance-based redundancy across 17 benchmarks, while SAE-guided subset selection produces more stable and representative subsets than random sampling. SAE feature overlap also predicts inter-benchmark performance similarity across 8 benchmarks. These results show that SAE features can turn benchmark analysis from a costly performance-matrix problem into a feature-footprint problem, enabling evaluation-free benchmark compression and suite curation.