2026
EMNLP
EMNLP 2026
Length and Meaning for Language Model Maps
Abstract
AbstractVarious studies have proposed methods to quantify and compare the rapidly growing number of large language models (LLMs). However, when constructing a macroscopic map of LLMs, there is a trade-off between estimation accuracy and computational cost, making the selection of evaluation texts critical. In this work, we propose Predictive Importance Sampling (PredIS), which selects informative texts by predicting likelihood variance from text-level features. Experiments on 100 LLMs across 25 domains demonstrate that our predictor generalizes robustly. Furthermore, we reveal that text importance cannot be explained by text length alone, as it is also closely tied to meaning reflected in the embedding space.