Anomalous State Sequence Modeling to Enhance Safety in Reinforcement Learning
This addresses safety concerns in AI deployment for high-cost domains, but it is incremental as it builds on existing safe RL methods with a new anomaly detection component.
The paper tackles the problem of ensuring safety in reinforcement learning for decision-making applications by proposing a novel approach that uses anomalous state sequences to detect unsafe states and train risk-averse policies, resulting in safer policies in safety-critical environments like self-driving cars.
The deployment of artificial intelligence (AI) in decision-making applications requires ensuring an appropriate level of safety and reliability, particularly in changing environments that contain a large number of unknown observations. To address this challenge, we propose a novel safe reinforcement learning (RL) approach that utilizes an anomalous state sequence to enhance RL safety. Our proposed solution Safe Reinforcement Learning with Anomalous State Sequences (AnoSeqs) consists of two stages. First, we train an agent in a non-safety-critical offline 'source' environment to collect safe state sequences. Next, we use these safe sequences to build an anomaly detection model that can detect potentially unsafe state sequences in a 'target' safety-critical environment where failures can have high costs. The estimated risk from the anomaly detection model is utilized to train a risk-averse RL policy in the target environment; this involves adjusting the reward function to penalize the agent for visiting anomalous states deemed unsafe by our anomaly model. In experiments on multiple safety-critical benchmarking environments including self-driving cars, our solution approach successfully learns safer policies and proves that sequential anomaly detection can provide an effective supervisory signal for training safety-aware RL agents