IxDRL: A Novel Explainable Deep Reinforcement Learning Toolkit based on Analyses of Interestingness
This addresses the explainability gap in deep reinforcement learning for critical applications where human oversight is needed, though it appears incremental as it builds on existing interestingness analysis concepts.
The paper tackles the problem of deep reinforcement learning systems lacking explainability mechanisms for human operators, proposing a novel toolkit called IxDRL that provides various measures of RL agent competence through interestingness analysis. The result is a framework that enables insights into agent capabilities and limitations, applicable to a wide range of RL algorithms and demonstrated in scenarios of varying complexity.
In recent years, advances in deep learning have resulted in a plethora of successes in the use of reinforcement learning (RL) to solve complex sequential decision tasks with high-dimensional inputs. However, existing systems lack the necessary mechanisms to provide humans with a holistic view of their competence, presenting an impediment to their adoption, particularly in critical applications where the decisions an agent makes can have significant consequences. Yet, existing RL-based systems are essentially competency-unaware in that they lack the necessary interpretation mechanisms to allow human operators to have an insightful, holistic view of their competency. Towards more explainable Deep RL (xDRL), we propose a new framework based on analyses of interestingness. Our tool provides various measures of RL agent competence stemming from interestingness analysis and is applicable to a wide range of RL algorithms, natively supporting the popular RLLib toolkit. We showcase the use of our framework by applying the proposed pipeline in a set of scenarios of varying complexity. We empirically assess the capability of the approach in identifying agent behavior patterns and competency-controlling conditions, and the task elements mostly responsible for an agent's competence, based on global and local analyses of interestingness. Overall, we show that our framework can provide agent designers with insights about RL agent competence, both their capabilities and limitations, enabling more informed decisions about interventions, additional training, and other interactions in collaborative human-machine settings.