Safe Reinforcement Learning with Chance-constrained Model Predictive Control
This addresses safety constraints for real-world RL applications, but it is incremental as it builds on existing MPC and policy gradient methods.
The paper tackles the challenge of ensuring safety in reinforcement learning by integrating a model predictive control guide with chance constraints into a policy gradient framework, achieving a provably safe optimal policy and demonstrating it on a simulated quadrotor.
Real-world reinforcement learning (RL) problems often demand that agents behave safely by obeying a set of designed constraints. We address the challenge of safe RL by coupling a safety guide based on model predictive control (MPC) with a modified policy gradient framework in a linear setting with continuous actions. The guide enforces safe operation of the system by embedding safety requirements as chance constraints in the MPC formulation. The policy gradient training step then includes a safety penalty which trains the base policy to behave safely. We show theoretically that this penalty allows for a provably safe optimal base policy and illustrate our method with a simulated linearized quadrotor experiment.