Counterfactual Explainer Framework for Deep Reinforcement Learning Models Using Policy Distillation
This addresses the need for robust verification techniques in safety-critical DRL applications, though it is incremental as it builds on existing counterfactual explanation methods.
The paper tackles the problem of explaining decisions made by black-box deep reinforcement learning models in safety-critical systems, proposing a counterfactual explanation framework that generates plausible and meaningful explanations, as demonstrated through experiments in automated driving and Atari Pong domains.
Deep Reinforcement Learning (DRL) has demonstrated promising capability in solving complex control problems. However, DRL applications in safety-critical systems are hindered by the inherent lack of robust verification techniques to assure their performance in such applications. One of the key requirements of the verification process is the development of effective techniques to explain the system functionality, i.e., why the system produces specific results in given circumstances. Recently, interpretation methods based on the Counterfactual (CF) explanation approach have been proposed to address the problem of explanation in DRLs. This paper proposes a novel CF explanation framework to explain the decisions made by a black-box DRL. To evaluate the efficacy of the proposed explanation framework, we carried out several experiments in the domains of automated driving systems and Atari Pong game. Our analysis demonstrates that the proposed framework generates plausible and meaningful explanations for various decisions made by deep underlying DRLs. Source codes are available at: \url{https://github.com/Amir-Samadi/Counterfactual-Explanation}