Counterfactual Explanations as Plans
This work addresses the need for more nuanced explainability in AI for applications involving sequences of actions, though it appears incremental in building on existing planning and explanation concepts.
The paper tackles the problem of providing richer explanations for AI systems in sequential decision-making contexts by formalizing counterfactual explanations as action sequences, and it shows that this approach naturally leads to model reconciliation methods for correcting user or agent models.
There has been considerable recent interest in explainability in AI, especially with black-box machine learning models. As correctly observed by the planning community, when the application at hand is not a single-shot decision or prediction, but a sequence of actions that depend on observations, a richer notion of explanations are desirable. In this paper, we look to provide a formal account of ``counterfactual explanations," based in terms of action sequences. We then show that this naturally leads to an account of model reconciliation, which might take the form of the user correcting the agent's model, or suggesting actions to the agent's plan. For this, we will need to articulate what is true versus what is known, and we appeal to a modal fragment of the situation calculus to formalise these intuitions. We consider various settings: the agent knowing partial truths, weakened truths and having false beliefs, and show that our definitions easily generalize to these different settings.