Understanding Syllogistic Reasoning in LLMs from Formal and Natural Language Perspectives
This work addresses the problem of understanding reasoning capabilities in LLMs for researchers in AI and cognitive science, but it is incremental as it builds on existing studies of logical reasoning in language models.
The study investigated syllogistic reasoning in 14 large language models, examining their capabilities in symbolic inferences and natural language understanding, and found that while reasoning is not uniform across models, some achieved perfect symbolic performance, raising questions about their role as formal reasoning mechanisms.
We study syllogistic reasoning in LLMs from the logical and natural language perspectives. In process, we explore fundamental reasoning capabilities of the LLMs and the direction this research is moving forward. To aid in our studies, we use 14 large language models and investigate their syllogistic reasoning capabilities in terms of symbolic inferences as well as natural language understanding. Even though this reasoning mechanism is not a uniform emergent property across LLMs, the perfect symbolic performances in certain models make us wonder whether LLMs are becoming more and more formal reasoning mechanisms, rather than making explicit the nuances of human reasoning.