Follow My Lead: Logical Fallacy Classification with Knowledge-Augmented LLMs
This addresses reasoning deficits in LLMs for applications requiring reliable logical analysis, though it is incremental as it builds on existing neuro-symbolic approaches.
The paper tackled the problem of LLMs' poor accuracy in logical fallacy classification by introducing a low-cost, instruction-based intervention with stepwise decomposition and knowledge graph verification, resulting in significant improvement in classification performance.
Large Language Models (LLMs) suffer from critical reasoning gaps, including a tendency to hallucinate and poor accuracy in classifying logical fallacies. This limitation stems from their default System 1 processing, which is fast and intuitive, whereas reliable reasoning requires the deliberate, effortful System 2 approach (Kahneman, 2011; Li et al., 2025). Since full System 2 training is often prohibitively expensive, we explore a low-cost, instruction-based intervention to bridge this gap. Our methodology introduces a novel stepwise instruction dataset that decomposes fallacy classification into a series of atomic procedural steps (simple binary questions). We further augment this with a final verification step where models consult a relational knowledge graph of related fallacies. This procedural, rule-based intervention yields a significant improvement in LLM logical fallacy classification. Crucially, the approach also provides enhanced transparency into the LLMs' decision-making, highlighting a practical pathway for Neuro-symbolic architectures to address LLM reasoning deficits.