Probabilistic Shielding for Safe Reinforcement Learning
This addresses the challenge of safe reinforcement learning for agents in real-life scenarios where safety constraints must be met, offering a scalable solution with formal guarantees.
The paper tackles the problem of ensuring strict safety guarantees in reinforcement learning by introducing a scalable method based on state-augmentation and shielding, which provides formal safety guarantees during both training and testing, as demonstrated through experimental evaluation.
In real-life scenarios, a Reinforcement Learning (RL) agent aiming to maximise their reward, must often also behave in a safe manner, including at training time. Thus, much attention in recent years has been given to Safe RL, where an agent aims to learn an optimal policy among all policies that satisfy a given safety constraint. However, strict safety guarantees are often provided through approaches based on linear programming, and thus have limited scaling. In this paper we present a new, scalable method, which enjoys strict formal guarantees for Safe RL, in the case where the safety dynamics of the Markov Decision Process (MDP) are known, and safety is defined as an undiscounted probabilistic avoidance property. Our approach is based on state-augmentation of the MDP, and on the design of a shield that restricts the actions available to the agent. We show that our approach provides a strict formal safety guarantee that the agent stays safe at training and test time. Furthermore, we demonstrate that our approach is viable in practice through experimental evaluation.