Bootstrapped Model Predictive Control
This work addresses a bottleneck in reinforcement learning for robotics and control systems, offering incremental improvements in MPC efficiency and stability.
The paper tackles the problem of poor policy learning and inaccurate value estimation in Model Predictive Control (MPC) for complex continuous control tasks by introducing Bootstrapped Model Predictive Control (BMPC), which uses bootstrapped policy learning and model-based TD-learning to improve data efficiency, asymptotic performance, and training stability, achieving superior results on high-dimensional locomotion tasks.
Model Predictive Control (MPC) has been demonstrated to be effective in continuous control tasks. When a world model and a value function are available, planning a sequence of actions ahead of time leads to a better policy. Existing methods typically obtain the value function and the corresponding policy in a model-free manner. However, we find that such an approach struggles with complex tasks, resulting in poor policy learning and inaccurate value estimation. To address this problem, we leverage the strengths of MPC itself. In this work, we introduce Bootstrapped Model Predictive Control (BMPC), a novel algorithm that performs policy learning in a bootstrapped manner. BMPC learns a network policy by imitating an MPC expert, and in turn, uses this policy to guide the MPC process. Combined with model-based TD-learning, our policy learning yields better value estimation and further boosts the efficiency of MPC. We also introduce a lazy reanalyze mechanism, which enables computationally efficient imitation learning. Our method achieves superior performance over prior works on diverse continuous control tasks. In particular, on challenging high-dimensional locomotion tasks, BMPC significantly improves data efficiency while also enhancing asymptotic performance and training stability, with comparable training time and smaller network sizes. Code is available at https://github.com/wertyuilife2/bmpc.