Detecting and Characterizing Planning in Language Models
This work addresses the need for reproducible mechanistic studies of planning in LLMs, providing a scalable foundation for researchers, though it is incremental as it builds on prior assumptions about planning.
The paper tackled the problem of distinguishing planning from improvisation in language models by developing formal criteria and a semi-automated annotation pipeline, finding that planning is not universal and varies across models and tasks, with Gemma-2-2B showing mixed behaviors on MBPP and improvisation in poem generation.
Modern large language models (LLMs) have demonstrated impressive performance across a wide range of multi-step reasoning tasks. Recent work suggests that LLMs may perform planning - selecting a future target token in advance and generating intermediate tokens that lead towards it - rather than merely improvising one token at a time. However, existing studies assume fixed planning horizons and often focus on single prompts or narrow domains. To distinguish planning from improvisation across models and tasks, we present formal and causally grounded criteria for detecting planning and operationalize them as a semi-automated annotation pipeline. We apply this pipeline to both base and instruction-tuned Gemma-2-2B models on the MBPP code generation benchmark and a poem generation task where Claude 3.5 Haiku was previously shown to plan. Our findings show that planning is not universal: unlike Haiku, Gemma-2-2B solves the same poem generation task through improvisation, and on MBPP it switches between planning and improvisation across similar tasks and even successive token predictions. We further show that instruction tuning refines existing planning behaviors in the base model rather than creating them from scratch. Together, these studies provide a reproducible and scalable foundation for mechanistic studies of planning in LLMs.