Beyond Memorization: Distinguishing between Reductive and Epistemic Reasoning in LLMs using Classic Logic Puzzles
This work addresses the challenge of evaluating reasoning capabilities in LLMs for AI researchers, though it is incremental in refining existing evaluation methods.
The paper tackled the problem of distinguishing between reductive reasoning and true epistemic reasoning in large language models (LLMs) by introducing a reduction ladder to modify classic logic puzzles, finding that models often rely on reduction rather than genuine reasoning and struggle with epistemic tasks.
Epistemic reasoning requires agents to infer the state of the world from partial observations and information about other agents' knowledge. Prior work evaluating LLMs on canonical epistemic puzzles interpreted their behavior through a dichotomy between epistemic reasoning and brittle memorization. We argue that this framing is incomplete: in recent models, memorization is better understood as a special case of reduction, where a new instance is mapped onto a known problem. Instead, we introduce a reduction ladder, a sequence of modifications that progressively move instances away from a canonical epistemic puzzle, making reduction increasingly difficult while preserving the underlying logic. We find that while some large models succeed via reduction, other models fail early, and all models struggle once epistemic reasoning is required.