Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores
Abstract
AbstractStandard reasoning benchmarks conflate two distinct failure modes into a single accuracy metric: absent internal answer representations versus flawed output sequence scoring. In this work, we show these failure modes can be disentangled. Across controlled logic, ProofWriter, ANLI, and FOLIO, we compare internal hidden-state probes against output candidate-answer string scores. We find that in post-trained language models, correct answers are often strongly encoded in internal representations even when raw sequence scoring collapses to chance or incorrect labels. To expose this hidden evidence, we apply a lightweight diagnostic correction: adjusting per-label output log-probabilities via two additive offsets fitted from as few as 25 unlabeled examples to match a prespecified label prior. This target-label-free correction recovers between 9 and 34 accuracy points (with matching macro-F1 gains) across larger Qwen3.5 checkpoints, and successfully recovers performance on OLMo-2 and Llama-3.1. The gains survive on examples a TF-IDF control misses and significantly exceed label-count permutation nulls. Our findings demonstrate that many LLM logical reasoning failures are output scoring artifacts rather than internal reasoning deficits, showing that benchmark accuracy can systematically underestimate a model’s representational capacity.