Language Models Do Not Follow Occam's Razor: A Benchmark for Inductive and Abductive Reasoning
This work addresses a gap in AI reasoning evaluation for real-world problem-solving, though it is incremental as it builds on existing LLM assessment frameworks.
The paper tackles the problem of evaluating large language models' inductive and abductive reasoning capabilities, introducing a synthetic dataset and a new metric based on Occam's Razor, and finds that LLMs struggle with complex scenarios and produce low-quality hypotheses despite using reasoning-enhancing techniques.
Reasoning is a core capability in artificial intelligence systems, for which large language models (LLMs) have recently shown remarkable progress. However, most work focuses exclusively on deductive reasoning, which is problematic since other types of reasoning are also essential in solving real-world problems, and they are less explored. This work focuses on evaluating LLMs' inductive and abductive reasoning capabilities. We introduce a programmable and synthetic dataset, InAbHyD (pronounced in-a-bid), where each reasoning example consists of an incomplete world model and a set of observations. The task for the intelligent agent is to produce hypotheses to explain observations under the incomplete world model to solve each reasoning example. We propose a new metric to evaluate the quality of hypotheses based on Occam's Razor. We evaluate and analyze some state-of-the-art LLMs. Our analysis shows that LLMs can perform inductive and abductive reasoning in simple scenarios, but struggle with complex world models and producing high-quality hypotheses, even with popular reasoning-enhancing techniques such as in-context learning and RLVR.