Generative Perturbation Analysis for Probabilistic Black-Box Anomaly Attribution
This work addresses the challenge of explaining anomalous deviations in black-box models for applications like fault detection, offering a probabilistic approach that is novel but incremental in the field of explainable AI.
The paper tackles the problem of probabilistic anomaly attribution in black-box regression, where the goal is to compute probability distributions for attribution scores of input variables given an anomaly, without access to training data. It proposes a novel framework that uses generative perturbations to quantify uncertainty, addressing the limitation of deviation-agnostic methods like Shapley values.
We address the task of probabilistic anomaly attribution in the black-box regression setting, where the goal is to compute the probability distribution of the attribution score of each input variable, given an observed anomaly. The training dataset is assumed to be unavailable. This task differs from the standard XAI (explainable AI) scenario, since we wish to explain the anomalous deviation from a black-box prediction rather than the black-box model itself. We begin by showing that mainstream model-agnostic explanation methods, such as the Shapley values, are not suitable for this task because of their ``deviation-agnostic property.'' We then propose a novel framework for probabilistic anomaly attribution that allows us to not only compute attribution scores as the predictive mean but also quantify the uncertainty of those scores. This is done by considering a generative process for perturbations that counter-factually bring the observed anomalous observation back to normalcy. We introduce a variational Bayes algorithm for deriving the distributions of per variable attribution scores. To the best of our knowledge, this is the first probabilistic anomaly attribution framework that is free from being deviation-agnostic.