Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Guided Permutation Scoring (GPS) : Option-Order Permutation for Multiple-Choice Question Answering

AbstractLarge Language Models (LLMs) are sensitive to the order in which answer choices are presented in Multiple-Choice Question Answering. The same question can yield a different prediction when its options are shuffled. Although exhaustive majority voting over all permutations can mitigate this bias, its factorial cost makes it impractical at inference time. We propose GPS (Guided Permutation Scoring), a two-stage inference-time pipeline that selects option orderings more likely to produce correct predictions from a frozen LLM. A sensitivity classifier first identifies which questions are genuinely vulnerable to positional interference; robust questions are answered directly, while sensitive ones are forwarded to a trained permutation scorer that identifies a more position-stable option ordering, promoting better predictions. Unlike prior approaches that either enumerate all permutations uniformly, or require modifying the LLM’s parameters, GPS is the first method to actively select permutations so that the base model can consistently give better answers. GPS steadily improves Accuracy, and Macro F1 across all evaluated model-dataset combinations, with remarkable reductions in positional bias metrics.

More from EMNLP 2026

All EMNLP 2026 papers →