Neuro-Symbolic Reasoning with External Inference Tools Mitigates Bias from Sub-Formula Level Logical Structure
Abstract
AbstractLogical reasoning is a core ability of modern language models, and various approaches have been proposed to enhance it. One effective method is neuro-symbolic approach, which makes a model output logic formulas and predict the answer based on them. However, fair comparison is challenging because existing methods use different experimental settings, such as post-processing and prompt styles. In addition, existing error analyses are too coarse to capture the impact of sub-formula level structures. In this paper, we present a systematic analysis of neuro-symbolic approaches under a unified setting with sub-formula level annotation. We reveal that some sub-formula level structures bias the model’s inference toward assuming the answer is true, and neuro-symbolic approaches with an external inference tool help mitigate this bias. On the other hand, these methods sometimes lack flexibility in representing sentence meaning as logic formulas.