Interpreting Emergent Planning in Model-Free Reinforcement Learning
This advances understanding of internal planning mechanisms in agents, which is important for emergent capabilities in large language models through reinforcement learning, though it is incremental as it builds on existing methods in interpretability.
The study tackled the problem of understanding whether model-free reinforcement learning agents can learn to plan, by applying concept-based interpretability to a model-free agent in Sokoban, and demonstrated that the agent uses learned concept representations to internally formulate plans that predict long-term effects and influence actions, with evidence including the emergence of a planning-like property benefiting from additional test-time compute.
We present the first mechanistic evidence that model-free reinforcement learning agents can learn to plan. This is achieved by applying a methodology based on concept-based interpretability to a model-free agent in Sokoban -- a commonly used benchmark for studying planning. Specifically, we demonstrate that DRC, a generic model-free agent introduced by Guez et al. (2019), uses learned concept representations to internally formulate plans that both predict the long-term effects of actions on the environment and influence action selection. Our methodology involves: (1) probing for planning-relevant concepts, (2) investigating plan formation within the agent's representations, and (3) verifying that discovered plans (in the agent's representations) have a causal effect on the agent's behavior through interventions. We also show that the emergence of these plans coincides with the emergence of a planning-like property: the ability to benefit from additional test-time compute. Finally, we perform a qualitative analysis of the planning algorithm learned by the agent and discover a strong resemblance to parallelized bidirectional search. Our findings advance understanding of the internal mechanisms underlying planning behavior in agents, which is important given the recent trend of emergent planning and reasoning capabilities in LLMs through RL