Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Decodability Is Not Computation: Examining Probing Claims

AbstractHigh probe accuracy does not by itself establish that a language model computes and uses the decoded value. We examine this inference in a syntax-sensitive arithmetic testbed: Llama-3.1-8B often fails multiplication when an operand is wrapped in parentheses or an additive identity. In the primary operand band, an input-only linear baseline nearly matches the original hidden-state probe’s regression R2 ≈ 0.96. A classical variance decomposition gives an analytical headroom of only 0.031 for this baseline class. In wider bands, selection-separated reanalysis instead finds positive activation advantages, which increase with regularization tuning; joint input-and-activation probes also improve over the baseline. These predictive gains are distinct from exact integer recovery and causal use. In the primary band, tested answer-site steering and state transplants produce small average output shifts and no observed donor-answer emissions, whereas operand-site interventions produce large effects. Operand access is necessary, with a strong bottleneck at one attention layer. The causal tests do not cover the wider-band predictive gains. We advocate explicit input baselines, matched causal tests, and positive controls, with conclusions restricted to the distributions, representations, and interventions actually examined.

More from EMNLP 2026

All EMNLP 2026 papers →