RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
This work addresses a limitation in LLMs' reasoning for AI researchers, providing a counterbalance to methods that rely on formal logic, though it is incremental as it focuses on dataset creation and evaluation rather than proposing a new solution.
The study tackled the problem of large language models (LLMs) failing to handle 'rulebreaker' scenarios where formal logic leads to conclusions that humans would reject based on common sense, by creating the RULEBREAKERS dataset for evaluation. The result showed that most models, including GPT-4o, achieved mediocre accuracy and tended to over-rigidly apply logical rules, unlike human reasoners.
Formal logic enables computers to reason in natural language by representing sentences in symbolic forms and applying rules to derive conclusions. However, in what our study characterizes as "rulebreaker" scenarios, this method can lead to conclusions that are typically not inferred or accepted by humans given their common sense and factual knowledge. Inspired by works in cognitive science, we create RULEBREAKERS, the first dataset for rigorously evaluating the ability of large language models (LLMs) to recognize and respond to rulebreakers (versus non-rulebreakers) in a human-like manner. Evaluating seven LLMs, we find that most models, including GPT-4o, achieve mediocre accuracy on RULEBREAKERS and exhibit some tendency to over-rigidly apply logical rules unlike what is expected from typical human reasoners. Further analysis suggests that this apparent failure is potentially associated with the models' poor utilization of their world knowledge and their attention distribution patterns. Whilst revealing a limitation of current LLMs, our study also provides a timely counterbalance to a growing body of recent works that propose methods relying on formal logic to improve LLMs' general reasoning capabilities, highlighting their risk of further increasing divergence between LLMs and human-like reasoning.