On the Limits of Innate Planning in Large Language Models
This work highlights substantial limitations in LLMs' innate planning abilities, which is a problem for AI researchers and developers seeking reliable autonomous reasoning systems, and it is incremental in building on existing critiques of LLM capabilities.
The study tackled the problem of assessing planning and stateful reasoning in large language models (LLMs) using the 8-puzzle task, finding that models failed to solve puzzles even with external move validation and exhibited deficits like brittle state representations and weak heuristic planning.
Large language models (LLMs) achieve impressive results on many benchmarks, yet their capacity for planning and stateful reasoning remains unclear. We study these abilities directly, without code execution or other tools, using the 8-puzzle: a classic task that requires state tracking and goal-directed planning while allowing precise, step-by-step evaluation. Four models are tested under common prompting conditions (Zero-Shot, Chain-of-Thought, Algorithm-of-Thought) and with tiered corrective feedback. Feedback improves success rates for some model-prompt combinations, but many successful runs are long, computationally expensive, and indirect. We then examine the models with an external move validator that provides only valid moves. Despite this level of assistance, none of the models solve any puzzles in this setting. Qualitative analysis reveals two dominant deficits across all models: (1) brittle internal state representations, leading to frequent invalid moves, and (2) weak heuristic planning, with models entering loops or selecting actions that do not reduce the distance to the goal state. These findings indicate that, in the absence of external tools such as code interpreters, current LLMs have substantial limitations in planning and that further progress may require mechanisms for maintaining explicit state and performing structured search.