Contrastive Explanations for Reinforcement Learning in terms of Expected Consequences
This addresses the need for transparency in RL models to improve trust and acceptance, though it is incremental as it builds on existing work in explainable AI.
The authors tackled the problem of understanding Reinforcement Learning (RL) models by proposing a method for RL agents to explain their behavior in terms of expected consequences of state transitions and outcomes, with results from a pilot survey indicating that users prefer explanations about policies over single actions.
Machine Learning models become increasingly proficient in complex tasks. However, even for experts in the field, it can be difficult to understand what the model learned. This hampers trust and acceptance, and it obstructs the possibility to correct the model. There is therefore a need for transparency of machine learning models. The development of transparent classification models has received much attention, but there are few developments for achieving transparent Reinforcement Learning (RL) models. In this study we propose a method that enables a RL agent to explain its behavior in terms of the expected consequences of state transitions and outcomes. First, we define a translation of states and actions to a description that is easier to understand for human users. Second, we developed a procedure that enables the agent to obtain the consequences of a single action, as well as its entire policy. The method calculates contrasts between the consequences of a policy derived from a user query, and of the learned policy of the agent. Third, a format for generating explanations was constructed. A pilot survey study was conducted to explore preferences of users for different explanation properties. Results indicate that human users tend to favor explanations about policy rather than about single actions.