Be Careful When Evaluating Explanations Regarding Ground Truth
This addresses the reliability of AI systems in safety-critical applications like medical imaging or robotics, though it is incremental in refining evaluation practices.
The paper tackles the problem that evaluating explanations of image classifiers using ground truth primarily assesses model quality rather than explanation methods, and proposes a framework to jointly evaluate the robustness of systems combining deep neural networks with explanation methods, finding vulnerabilities in vision transformers and AI systems to adversarial attacks.
Evaluating explanations of image classifiers regarding ground truth, e.g. segmentation masks defined by human perception, primarily evaluates the quality of the models under consideration rather than the explanation methods themselves. Driven by this observation, we propose a framework for $\textit{jointly}$ evaluating the robustness of safety-critical systems that $\textit{combine}$ a deep neural network with an explanation method. These are increasingly used in real-world applications like medical image analysis or robotics. We introduce a fine-tuning procedure to (mis)align model$\unicode{x2013}$explanation pipelines with ground truth and use it to quantify the potential discrepancy between worst and best-case scenarios of human alignment. Experiments across various model architectures and post-hoc local interpretation methods provide insights into the robustness of vision transformers and the overall vulnerability of such AI systems to potential adversarial attacks.