Advantage Actor-Critic with Reasoner: Explaining the Agent's Behavior from an Exploratory Perspective
This addresses the problem of interpretability in RL for domains with real-world consequences, offering incremental improvements by integrating a reasoner into existing Actor-Critic frameworks.
The paper tackles the lack of transparency in reinforcement learning by proposing Advantage Actor-Critic with Reasoner (A2CR), which adds a Reasoner Network to Actor-Critic models to generate interpretable explanations of agent behavior, with evaluations in Super Mario Bros showing changes in predicted label proportions and more focused saliencies as exploration intensifies.
Reinforcement learning (RL) is a powerful tool for solving complex decision-making problems, but its lack of transparency and interpretability has been a major challenge in domains where decisions have significant real-world consequences. In this paper, we propose a novel Advantage Actor-Critic with Reasoner (A2CR), which can be easily applied to Actor-Critic-based RL models and make them interpretable. A2CR consists of three interconnected networks: the Policy Network, the Value Network, and the Reasoner Network. By predefining and classifying the underlying purpose of the actor's actions, A2CR automatically generates a more comprehensive and interpretable paradigm for understanding the agent's decision-making process. It offers a range of functionalities such as purpose-based saliency, early failure detection, and model supervision, thereby promoting responsible and trustworthy RL. Evaluations conducted in action-rich Super Mario Bros environments yield intriguing findings: Reasoner-predicted label proportions decrease for ``Breakout" and increase for ``Hovering" as the exploration level of the RL algorithm intensifies. Additionally, purpose-based saliencies are more focused and comprehensible.