What Did You Think Would Happen? Explaining Agent Behaviour Through Intended Outcomes
This addresses the need for better interpretability in reinforcement learning, though it is incremental as it builds on existing Q-function methods.
The paper tackles the problem of explaining reinforcement learning agent behavior by introducing a novel explanation method based on intended outcomes, and it demonstrates this approach on multiple RL problems with code provided for researchers.
We present a novel form of explanation for Reinforcement Learning, based around the notion of intended outcome. These explanations describe the outcome an agent is trying to achieve by its actions. We provide a simple proof that general methods for post-hoc explanations of this nature are impossible in traditional reinforcement learning. Rather, the information needed for the explanations must be collected in conjunction with training the agent. We derive approaches designed to extract local explanations based on intention for several variants of Q-function approximation and prove consistency between the explanations and the Q-values learned. We demonstrate our method on multiple reinforcement learning problems, and provide code to help researchers introspecting their RL environments and algorithms.