Dependency Decomposition and a Reject Option for Explainable Models
This research addresses the need for explainable, robust, and uncertainty-aware machine learning models for safety-critical domains like autonomous driving and medical diagnosis, offering an incremental step towards more trustworthy AI.
This paper investigates the dependencies between probability distributions of image classification outputs and their explaining variables (e.g., attributes, texts, heatmaps) in deep learning models. They propose an Explanation Dependency Decomposition (EDD) and use the resulting explanation to verify (accept or reject) the prediction.
Deploying machine learning models in safety-related do-mains (e.g. autonomous driving, medical diagnosis) demands for approaches that are explainable, robust against adversarial attacks and aware of the model uncertainty. Recent deep learning models perform extremely well in various inference tasks, but the black-box nature of these approaches leads to a weakness regarding the three requirements mentioned above. Recent advances offer methods to visualize features, describe attribution of the input (e.g.heatmaps), provide textual explanations or reduce dimensionality. However,are explanations for classification tasks dependent or are they independent of each other? For in-stance, is the shape of an object dependent on the color? What is the effect of using the predicted class for generating explanations and vice versa? In the context of explainable deep learning models, we present the first analysis of dependencies regarding the probability distribution over the desired image classification outputs and the explaining variables (e.g. attributes, texts, heatmaps). Therefore, we perform an Explanation Dependency Decomposition (EDD). We analyze the implications of the different dependencies and propose two ways of generating the explanation. Finally, we use the explanation to verify (accept or reject) the prediction