Skip to content
conftrace_
2026 EMNLP EMNLP 2026

A Hyperbolicity Atlas of Large Language Model Hidden States

AbstractHidden states of large language models (LLMs) are typically analyzed as flat Euclidean vectors through probing, anisotropy, and intrinsic-dimension lenses. Yet much of what these models process (taxonomies, program structure, multi-step derivations) is hierarchical, and whether hidden-state distances reflect such hierarchy remains untested. We present the first systematic study of Gromov hyperbolicity (GH), a distance-based measure of tree-likeness, in prompt-token hidden states of open-weight LLMs. Across model families, scales, layer depths, and domains spanning math, code, commonsense, and truthfulness benchmarks, we construct a hyperbolicity atlas charting the normalized statistic 𝛿rel along each axis. Four findings emerge: (i) depth dominates, with middle layers forming a high-𝛿rel plateau and final layers dropping toward tree-like structure; (ii) scale is weak and non-monotonic; (iii) matched-scale families occupy distinct regimes, with specialization gaps exceeding scale effects; and (iv) domains interact with specialization, e.g., HumanEval shifts from the highest- to the lowest-𝛿rel domain between base and coder checkpoints. GH thus serves as a lightweight diagnostic that localizes where hierarchical distance structure emerges and which comparisons warrant deeper analysis.

More from EMNLP 2026

All EMNLP 2026 papers →