Avoidance Learning Using Observational Reinforcement Learning
This addresses safety concerns in real-world imitation learning where perfect experts are unavailable, though it appears incremental as it builds on existing policy optimization and safety methods.
The paper tackles the problem of learning to avoid dangerous behaviors from a demonstrator in reinforcement learning, presenting a framework that uses distance between state occupancy distributions as a reward bonus, and shows improved sample efficiency compared to state-of-the-art methods in experiments.
Imitation learning seeks to learn an expert policy from sampled demonstrations. However, in the real world, it is often difficult to find a perfect expert and avoiding dangerous behaviors becomes relevant for safety reasons. We present the idea of \textit{learning to avoid}, an objective opposite to imitation learning in some sense, where an agent learns to avoid a demonstrator policy given an environment. We define avoidance learning as the process of optimizing the agent's reward while avoiding dangerous behaviors given by a demonstrator. In this work we develop a framework of avoidance learning by defining a suitable objective function for these problems which involves the \emph{distance} of state occupancy distributions of the expert and demonstrator policies. We use density estimates for state occupancy measures and use the aforementioned distance as the reward bonus for avoiding the demonstrator. We validate our theory with experiments using a wide range of partially observable environments. Experimental results show that we are able to improve sample efficiency during training compared to state of the art policy optimization and safety methods.