Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings
This addresses the challenge of deploying RL in safety-critical applications like autonomous driving, offering a cautious adaptation approach that reduces failure risks, though it is incremental in building on existing model-based RL methods.
The paper tackles the problem of safely adapting reinforcement learning agents from simulated source environments to real-world safety-critical target settings by proposing CARL, which uses model-based RL to estimate risk and plan cautiously, resulting in higher rewards and fewer failures across multiple domains.
Reinforcement learning (RL) in real-world safety-critical target settings like urban driving is hazardous, imperiling the RL agent, other agents, and the environment. To overcome this difficulty, we propose a "safety-critical adaptation" task setting: an agent first trains in non-safety-critical "source" environments such as in a simulator, before it adapts to the target environment where failures carry heavy costs. We propose a solution approach, CARL, that builds on the intuition that prior experience in diverse environments equips an agent to estimate risk, which in turn enables relative safety through risk-averse, cautious adaptation. CARL first employs model-based RL to train a probabilistic model to capture uncertainty about transition dynamics and catastrophic states across varied source environments. Then, when exploring a new safety-critical environment with unknown dynamics, the CARL agent plans to avoid actions that could lead to catastrophic states. In experiments on car driving, cartpole balancing, half-cheetah locomotion, and robotic object manipulation, CARL successfully acquires cautious exploration behaviors, yielding higher rewards with fewer failures than strong RL adaptation baselines. Website at https://sites.google.com/berkeley.edu/carl.