The Surprising Difficulty of Search in Model-Based Reinforcement Learning
This work addresses a key problem in model-based RL for researchers and practitioners by revealing that search is not plug-and-play, which could improve algorithm design and performance.
The paper challenges the conventional view that long-term predictions and compounding errors are the primary obstacles in model-based reinforcement learning, showing that search can harm performance even with accurate models and that mitigating distribution shift is more critical, leading to state-of-the-art results on multiple benchmarks.
This paper investigates search in model-based reinforcement learning (RL). Conventional wisdom holds that long-term predictions and compounding errors are the primary obstacles for model-based RL. We challenge this view, showing that search is not a plug-and-play replacement for a learned policy. Surprisingly, we find that search can harm performance even when the model is highly accurate. Instead, we show that mitigating distribution shift matters more than improving model or value function accuracy. Building on this insight, we identify key techniques for enabling effective search, achieving state-of-the-art performance across multiple popular benchmark domains.