Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Directional Drift in Arithmetic Reasoning: How Comparative Cues Steer LLM Predictions

AbstractArithmetic tasks are widely used to evaluate LLM reasoning because they provide objective truth. Yet accuracy alone cannot distinguish whether prompt sensitivity reflects random prediction instability or systematic semantic bias. We study this question in comparative arithmetic, where semantically equivalent framings can cue directional outcomes such as more, less, or equal. We introduce MathComp, a balanced benchmark of 10,800 controlled comparison problems spanning multiple framing conditions and answer formats. Across ten LLMs, semantically equivalent comparative cues systematically bias predictions toward cued outcomes even when it is numerically incorrect. Notably, this effect persists even when cue words are removed from the answer options, suggesting that framing bias reflects broader semantic steering rather than simple label copying. Chain-of-thought reasoning mitigates the effect substantially, but residual directional drift remains. These findings suggest that prompt sensitivity in arithmetic reasoning can exhibit structured semantic bias toward linguistically cued outcomes, with implications for robustness evaluation and benchmark design.

More from EMNLP 2026

All EMNLP 2026 papers →