When Agreement Is Not Enough: Evidence-Risk Arbitration for Multi-Strategy Agentic QA
Abstract
AbstractMulti-strategy question-answering agents often produce several plausible responses to one query, but must ultimately show the user only one. Voting treats answer agreement as trust, yet strategies can disagree for different reasons, and identical answers can rest on evidence of unequal quality. We cast this final choice as diagnosis-first evidence-risk arbitration and introduce the Evidence-Risk-Aware Strategy Reranker (ERSR), which canonicalizes a pool of six responses, classifies their agreement type, and applies a type-specific evidence rule while returning one unchanged response. Calibration uses only development data; for each evaluation item, trusted reference answers, trusted reference evidence, and utility stay hidden, and the selector never abstains. On a 900-item development split drawn from FEVER, HotpotQA, and SciFact, ERSR improves evidence-aware utility over a development-selected fixed strategy by +0.052, with the largest gains on label conflicts and unanimous answers with uneven evidence. These results suggest that response-set structure, not agreement alone, is a useful signal for choosing trustworthy answers.