ProSh: Probabilistic Shielding for Model-free Reinforcement Learning
This addresses safety concerns for deploying RL systems in real-world applications, offering a novel approach with theoretical guarantees, though it builds on existing constrained MDP frameworks.
The paper tackles the problem of ensuring safety in reinforcement learning by introducing ProSh, a model-free algorithm that provides formal safety guarantees through probabilistic shielding and risk augmentation, achieving safe actions in expectation and preserving optimality in deterministic environments.
Safety is a major concern in reinforcement learning (RL): we aim at developing RL systems that not only perform optimally, but are also safe to deploy by providing formal guarantees about their safety. To this end, we introduce Probabilistic Shielding via Risk Augmentation (ProSh), a model-free algorithm for safe reinforcement learning under cost constraints. ProSh augments the Constrained MDP state space with a risk budget and enforces safety by applying a shield to the agent's policy distribution using a learned cost critic. The shield ensures that all sampled actions remain safe in expectation. We also show that optimality is preserved when the environment is deterministic. Since ProSh is model-free, safety during training depends on the knowledge we have acquired about the environment. We provide a tight upper-bound on the cost in expectation, depending only on the backup-critic accuracy, that is always satisfied during training. Under mild, practically achievable assumptions, ProSh guarantees safety even at training time, as shown in the experiments.