Agreement Is Not Truth: Characterizing and Controlling Consistent-but-Wrong Reasoning in LLMs
Abstract
AbstractSelf-consistency treats agreement among sampled chains of thought as evidence that the answer is correct. We show that this reading fails predictably. Across 5 reasoning datasets and 5 generators, consistent-but-wrong (CBW) rates track the task rather than the generator, and the pattern holds across agreement thresholds and sample budgets. On StrategyQA, at least 15% of answers are wrong at 90% self-agreement. We prove that no procedure reading only the answer distribution can always separate a question whose majority is right from one whose majority is wrong. We propose Risk-Controlled Self-Consistency (RISC), which combines the majority ratio with a verifier signal and calibrates an acceptance threshold by Learn-then-Test. It certifies the risk target in domain at finite sample size, and accepts nothing when no threshold qualifies. Against self-consistency alone, pooled selective error falls 3.01pp at 0.50 coverage; per-task gains are mixed. At 0.9 coverage, a local generator with a lightweight verifier costs ~9× less than an all-API baseline and shows no detectable selective-error gap on four of five datasets. It trails on MATH500. Higher agreement is not always higher accuracy. Calibration narrows the gap, but more sampling usually does not.