Counterfactual Explanations in Sequential Decision Making Under Uncertainty
This work addresses the need for interpretable AI in sequential settings, such as therapy, by extending counterfactual explanations beyond one-step decisions, though it is incremental as it builds on existing MDP and causal model frameworks.
The paper tackles the problem of generating counterfactual explanations for sequential decision-making processes, where multiple dependent actions occur over time, by introducing a polynomial-time dynamic programming algorithm that guarantees optimal explanations and validating it with synthetic and real data from cognitive behavioral therapy.
Methods to find counterfactual explanations have predominantly focused on one step decision making processes. In this work, we initiate the development of methods to find counterfactual explanations for decision making processes in which multiple, dependent actions are taken sequentially over time. We start by formally characterizing a sequence of actions and states using finite horizon Markov decision processes and the Gumbel-Max structural causal model. Building upon this characterization, we formally state the problem of finding counterfactual explanations for sequential decision making processes. In our problem formulation, the counterfactual explanation specifies an alternative sequence of actions differing in at most k actions from the observed sequence that could have led the observed process realization to a better outcome. Then, we introduce a polynomial time algorithm based on dynamic programming to build a counterfactual policy that is guaranteed to always provide the optimal counterfactual explanation on every possible realization of the counterfactual environment dynamics. We validate our algorithm using both synthetic and real data from cognitive behavioral therapy and show that the counterfactual explanations our algorithm finds can provide valuable insights to enhance sequential decision making under uncertainty.