Puzzle Solving using Reasoning of Large Language Models: A Survey
This survey addresses the problem of understanding LLMs' limitations in logical reasoning for AI researchers, but it is incremental as it synthesizes existing work without new experimental results.
The survey examines the capabilities of Large Language Models (LLMs) in solving puzzles, categorizing them into rule-based and rule-less types, and finds significant challenges in complex scenarios, highlighting a disparity with human-like reasoning.
Exploring the capabilities of Large Language Models (LLMs) in puzzle solving unveils critical insights into their potential and challenges in AI, marking a significant step towards understanding their applicability in complex reasoning tasks. This survey leverages a unique taxonomy -- dividing puzzles into rule-based and rule-less categories -- to critically assess LLMs through various methodologies, including prompting techniques, neuro-symbolic approaches, and fine-tuning. Through a critical review of relevant datasets and benchmarks, we assess LLMs' performance, identifying significant challenges in complex puzzle scenarios. Our findings highlight the disparity between LLM capabilities and human-like reasoning, particularly in those requiring advanced logical inference. The survey underscores the necessity for novel strategies and richer datasets to advance LLMs' puzzle-solving proficiency and contribute to AI's logical reasoning and creative problem-solving advancements.