Two-step counterfactual generation for OOD examples
This addresses the need for explainable AI in safety-critical systems by providing explanations for OOD predictions, though it appears incremental as it builds on existing OOD detection and explainability work.
The paper tackles the problem of explaining why a model predicts data as out-of-distribution (OOD) by introducing OOD counterfactuals, which are perturbed data points that move between OOD categories, and proposes a method for generating them, showing results compared to benchmarks on synthetic and benchmark data.
Two fundamental requirements for the deployment of machine learning models in safety-critical systems are to be able to detect out-of-distribution (OOD) data correctly and to be able to explain the prediction of the model. Although significant effort has gone into both OOD detection and explainable AI, there has been little work on explaining why a model predicts a certain data point is OOD. In this paper, we address this question by introducing the concept of an OOD counterfactual, which is a perturbed data point that iteratively moves between different OOD categories. We propose a method for generating such counterfactuals, investigate its application on synthetic and benchmark data, and compare it to several benchmark methods using a range of metrics.