Approximate Shielding of Atari Agents for Safe Exploration
This work addresses safe exploration for reinforcement learning agents in constrained settings like Atari games, but it is incremental as it builds on prior latent shielding methods with added features.
The paper tackled the problem of safe exploration in reinforcement learning for real-world tasks by proposing an approximate shielding algorithm that reduces safety violations in Atari games, with preliminary results showing effective reduction in violation rates and sometimes improved convergence speed and agent quality.
Balancing exploration and conservatism in the constrained setting is an important problem if we are to use reinforcement learning for meaningful tasks in the real world. In this paper, we propose a principled algorithm for safe exploration based on the concept of shielding. Previous approaches to shielding assume access to a safety-relevant abstraction of the environment or a high-fidelity simulator. Instead, our work is based on latent shielding - another approach that leverages world models to verify policy roll-outs in the latent space of a learned dynamics model. Our novel algorithm builds on this previous work, using safety critics and other additional features to improve the stability and farsightedness of the algorithm. We demonstrate the effectiveness of our approach by running experiments on a small set of Atari games with state dependent safety labels. We present preliminary results that show our approximate shielding algorithm effectively reduces the rate of safety violations, and in some cases improves the speed of convergence and quality of the final agent.