Perceive-Then-Adjudicate: A Multi-Agent Framework for Resolving Knowledge-Evidence Conflicts in LLM Fact-Checking
Abstract
AbstractFact-checking has transitioned from traditional semantic matching toward active reasoning powered by Large Language Models (LLMs), manifesting as the interplay between internal parametric knowledge and external grounded evidence. However, this paradigm introduces two critical challenges: (i) excessive reliance on external evidence even when it conflicts with correct internal judgment, and (ii) uncritical acceptance of a consensus even when both sources are erroneous. To address these limitations, we propose a Multi-Agent Arbitration Framework where a meta-cognitive Adjudication Agent (AA) orchestrates a Knowledge Agent (KA) and an Evidence Agent (EA). Through hybrid reinforcement and imitation learning, the AA perceives conflict signals between KA and EA to adaptively calibrate judgmental autonomy against evidentiary support. Upon detecting a potentially erroneous consensus, the AA invokes a Reflective Rejection mechanism to scrutinize and disrupt deceptive alignment. Empirical evaluations on Snopes and PolitiFact demonstrate that our framework consistently outperforms state-of-the-art baselines. Notably, it effectively mitigates over-reliance errors while maintaining high fidelity in correct-consensus scenarios, recovering 26.2% of dual-error cases where static systems typically succumb to superficial consistency. Our code is publicly available at https://github.com/Zhang-Gorden/Perceive-Then-Adjudicate.