Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Lost in Choice: Evaluating LLMs Under Extreme Multiple-Choice Scaling

AbstractExisting multiple-choice evaluations use small option sets and thus conflate several sources of failure when candidate spaces grow. We provide a controlled factorial decomposition of large-choice degradation across four axes: whether a question requires comparing its options, candidate cardinality (up to 1,000), spatial layout, and gold-answer position. Across six reasoning LLMs, comparative questions degrade far more than questions solvable without inspecting the options; dispersion amplifies this gap; and accuracy often collapses at late-middle and terminal positions rather than recovering in the classic lost-in-the-middle U-shape. We call this asymmetric, structure-dependent failure mode Lost in Choice. A replication on 500 real ToolBench tool-selection trajectories per condition reproduces the same hardest position on two additional model-task evaluations. Finally, an audit of reasoning traces yields a taxonomy of the heuristics and shortcuts models adopt as candidate spaces expand.

More from EMNLP 2026

All EMNLP 2026 papers →