Experiential Explanations for Reinforcement Learning
This addresses the challenge of interpretability in RL for non-experts, enabling better understanding and intervention in AI decisions, though it is an incremental improvement over existing explanation methods.
The paper tackles the problem of making reinforcement learning (RL) systems interpretable for non-AI experts by proposing Experiential Explanations, a technique that generates counterfactual explanations using influence predictors trained alongside the RL policy. Human evaluation studies showed that participants with these explanations were better at guessing agent actions and rated them as more understandable, satisfying, complete, useful, and accurate compared to standard explanations.
Reinforcement learning (RL) systems can be complex and non-interpretable, making it challenging for non-AI experts to understand or intervene in their decisions. This is due in part to the sequential nature of RL in which actions are chosen because of their likelihood of obtaining future rewards. However, RL agents discard the qualitative features of their training, making it difficult to recover user-understandable information for "why" an action is chosen. We propose a technique Experiential Explanations to generate counterfactual explanations by training influence predictors along with the RL policy. Influence predictors are models that learn how different sources of reward affect the agent in different states, thus restoring information about how the policy reflects the environment. Two human evaluation studies revealed that participants presented with Experiential Explanations were better able to correctly guess what an agent would do than those presented with other standard types of explanation. Participants also found that Experiential Explanations are more understandable, satisfying, complete, useful, and accurate. Qualitative analysis provides information on the factors of Experiential Explanations that are most useful and the desired characteristics that participants seek from the explanations.