Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification
It provides a systematic classification for researchers and practitioners in control and AI, but it is incremental as it surveys and organizes existing work rather than introducing new methods.
This survey categorizes and analyzes the differences, similarities, and combination methods between Model Predictive Control (MPC) and Reinforcement Learning (RL), focusing on how MPC's online optimization can enhance RL's closed-loop policy performance.
The fields of MPC and RL consider two successful control techniques for Markov decision processes. Both approaches are derived from similar fundamental principles, and both are widely used in practical applications, including robotics, process control, energy systems, and autonomous driving. Despite their similarities, MPC and RL follow distinct paradigms that emerged from diverse communities and different requirements. Various technical discrepancies, particularly the role of an environment model as part of the algorithm, lead to methodologies with nearly complementary advantages. Due to their orthogonal benefits, research interest in combination methods has recently increased significantly, leading to a large and growing set of complex ideas leveraging MPC and RL. This work illuminates the differences, similarities, and fundamentals that allow for different combination algorithms and categorizes existing work accordingly. Particularly, we focus on the versatile actor-critic RL approach as a basis for our categorization and examine how the online optimization approach of MPC can be used to improve the overall closed-loop performance of a policy.