Hybrid Models for Natural Language Reasoning: The Case of Syllogistic Logic
This work addresses the problem of improving logical reasoning capabilities in AI systems, particularly for applications requiring generalization, by proposing a hybrid model that overcomes key barriers in neural reasoning, though it is incremental in nature.
The study tackled the challenge of generalization in neural models for logical reasoning by distinguishing between compositionality and recursiveness, finding that large language models (LLMs) struggle with compositionality while showing proficiency in recursiveness. To address this, a hybrid architecture combining symbolic reasoning and neural computation was proposed, achieving robust and efficient inference with preserved high efficiency even with small neural components.
Despite the remarkable progress in neural models, their ability to generalize, a cornerstone for applications like logical reasoning, remains a critical challenge. We delineate two fundamental aspects of this ability: compositionality, the capacity to abstract atomic logical rules underlying complex inferences, and recursiveness, the aptitude to build intricate representations through iterative application of inference rules. In the literature, these two aspects are often confounded together under the umbrella term of generalization. To sharpen this distinction, we investigated the logical generalization capabilities of pre-trained large language models (LLMs) using the syllogistic fragment as a benchmark for natural language reasoning. Though simple, this fragment provides a foundational yet expressive subset of formal logic that supports controlled evaluation of essential reasoning abilities. Our findings reveal a significant disparity: while LLMs demonstrate reasonable proficiency in recursiveness, they struggle with compositionality. To overcome these limitations and establish a reliable logical prover, we propose a hybrid architecture integrating symbolic reasoning with neural computation. This synergistic interaction enables robust and efficient inference, neural components accelerate processing, while symbolic reasoning ensures completeness. Our experiments show that high efficiency is preserved even with relatively small neural components. As part of our proposed methodology, this analysis gives a rationale and highlights the potential of hybrid models to effectively address key generalization barriers in neural reasoning systems.