Skip to content
conftrace_
2026 EMNLP EMNLP 2026

From Position Bias to Position-Stable Agreement: Testing What LLM Judges Evaluate

AbstractPairwise LLM-as-judge evaluation should preserve response identity when presentation order changes. Across four judge models, short A/B prompts yield 70–100% second-position selection (72/80 comparisons overall). Targeted prompt ablations show format sensitivity: label changes alter parsed choices, and a neutral buffer produces complete first-position selection on 59 fixed-order comparisons, reversing rather than removing position preference. We use a position-swapped evaluation protocol, retaining a directional verdict only when both orders select the same response. In a new 16-condition GPT-4.1 factorial audit, analysis-first and a verdict prefix raise directional coverage by 5.6 and 6.5 percentage points, respectively. On 40 Arena-Hard queries with three generators, the 120 pairs comprise 62 directional results, no double ties, 14 swap failures, and 44 invalid pairs. Cross-Model Agreement (CMA) yields 25 unique query winners. A retrospective domain-routing check predicts 17 of these 25 labels using rules fitted outside each test fold. These labels and any oracle score remain judge-relative; human-preference validity and downstream training gains are not established.

More from EMNLP 2026

All EMNLP 2026 papers →