Learning Safe Policies with Expert Guidance
This addresses safety concerns in RL for applications like robotics or autonomous systems, but it is incremental as it builds on existing expert-guided methods.
The paper tackles the problem of ensuring safe behavior in reinforcement learning when the reward function is hard to specify by using expert demonstrations, resulting in an agent that safely avoids negative states while imitating expert behavior in others.
We propose a framework for ensuring safe behavior of a reinforcement learning agent when the reward function may be difficult to specify. In order to do this, we rely on the existence of demonstrations from expert policies, and we provide a theoretical framework for the agent to optimize in the space of rewards consistent with its existing knowledge. We propose two methods to solve the resulting optimization: an exact ellipsoid-based method and a method in the spirit of the "follow-the-perturbed-leader" algorithm. Our experiments demonstrate the behavior of our algorithm in both discrete and continuous problems. The trained agent safely avoids states with potential negative effects while imitating the behavior of the expert in the other states.