Leveraging Approximate Model-based Shielding for Probabilistic Safety Guarantees in Continuous Environments
This work addresses safety in reinforcement learning for continuous settings, offering incremental improvements to existing shielding methods.
The paper tackles the challenge of applying shielding for safe reinforcement learning in continuous environments by extending the approximate model-based shielding framework, achieving strong probabilistic safety guarantees and proposing novel penalty techniques for stable convergence.
Shielding is a popular technique for achieving safe reinforcement learning (RL). However, classical shielding approaches come with quite restrictive assumptions making them difficult to deploy in complex environments, particularly those with continuous state or action spaces. In this paper we extend the more versatile approximate model-based shielding (AMBS) framework to the continuous setting. In particular we use Safety Gym as our test-bed, allowing for a more direct comparison of AMBS with popular constrained RL algorithms. We also provide strong probabilistic safety guarantees for the continuous setting. In addition, we propose two novel penalty techniques that directly modify the policy gradient, which empirically provide more stable convergence in our experiments.