Directed Policy Gradient for Safe Reinforcement Learning with Human Advice
This addresses safety and efficiency issues for RL agents interacting with humans, such as in workplaces or user settings, though it is an incremental improvement over existing policy gradient methods.
The paper tackles the problem of unsafe reinforcement learning agents in human-shared environments by proposing Directed Policy Gradient (DPG), which allows a teacher to override the agent to prevent undesirable actions and leverage human advice for faster learning. Experiments show that DPG learns much faster than reward-based approaches and requires an order of magnitude less advice.
Many currently deployed Reinforcement Learning agents work in an environment shared with humans, be them co-workers, users or clients. It is desirable that these agents adjust to people's preferences, learn faster thanks to their help, and act safely around them. We argue that most current approaches that learn from human feedback are unsafe: rewarding or punishing the agent a-posteriori cannot immediately prevent it from wrong-doing. In this paper, we extend Policy Gradient to make it robust to external directives, that would otherwise break the fundamentally on-policy nature of Policy Gradient. Our technique, Directed Policy Gradient (DPG), allows a teacher or backup policy to override the agent before it acts undesirably, while allowing the agent to leverage human advice or directives to learn faster. Our experiments demonstrate that DPG makes the agent learn much faster than reward-based approaches, while requiring an order of magnitude less advice.