Right for the Wrong Reason: Can Interpretable ML Techniques Detect Spurious Correlations?
This addresses the issue of unreliable model explanations for practitioners in medical AI, though it is incremental as it evaluates existing methods rather than proposing new ones.
The paper tackled the problem of whether interpretable ML techniques can detect spurious correlations in deep neural networks, finding that SHAP and Attri-Net performed best in identifying faulty model behavior in a chest x-ray diagnosis task.
While deep neural network models offer unmatched classification performance, they are prone to learning spurious correlations in the data. Such dependencies on confounding information can be difficult to detect using performance metrics if the test data comes from the same distribution as the training data. Interpretable ML methods such as post-hoc explanations or inherently interpretable classifiers promise to identify faulty model reasoning. However, there is mixed evidence whether many of these techniques are actually able to do so. In this paper, we propose a rigorous evaluation strategy to assess an explanation technique's ability to correctly identify spurious correlations. Using this strategy, we evaluate five post-hoc explanation techniques and one inherently interpretable method for their ability to detect three types of artificially added confounders in a chest x-ray diagnosis task. We find that the post-hoc technique SHAP, as well as the inherently interpretable Attri-Net provide the best performance and can be used to reliably identify faulty model behavior.