ActSafe: Active Exploration with Safety Constraints for Reinforcement Learning
This addresses the challenge of enabling RL agents to learn safely in real-world settings, which is crucial for applications like robotics and autonomous systems, though it builds incrementally on existing model-based RL methods.
The paper tackles the problem of unsafe exploration in reinforcement learning by introducing ActSafe, a model-based RL algorithm that ensures safety during learning and achieves near-optimal policies in finite time, with empirical results showing state-of-the-art performance on safe deep RL benchmarks.
Reinforcement learning (RL) is ubiquitous in the development of modern AI systems. However, state-of-the-art RL agents require extensive, and potentially unsafe, interactions with their environments to learn effectively. These limitations confine RL agents to simulated environments, hindering their ability to learn directly in real-world settings. In this work, we present ActSafe, a novel model-based RL algorithm for safe and efficient exploration. ActSafe learns a well-calibrated probabilistic model of the system and plans optimistically w.r.t. the epistemic uncertainty about the unknown dynamics, while enforcing pessimism w.r.t. the safety constraints. Under regularity assumptions on the constraints and dynamics, we show that ActSafe guarantees safety during learning while also obtaining a near-optimal policy in finite time. In addition, we propose a practical variant of ActSafe that builds on latest model-based RL advancements and enables safe exploration even in high-dimensional settings such as visual control. We empirically show that ActSafe obtains state-of-the-art performance in difficult exploration tasks on standard safe deep RL benchmarks while ensuring safety during learning.