Do Not Trust Additive Explanations
This addresses a critical problem for users of explainable AI who rely on these methods for trust in complex models, highlighting a significant limitation in current practices.
The paper investigates the faithfulness of additive explanation methods like LIME and SHAP for non-additive models, finding that they can be misleading due to interactions, and introduces a new method to detect such interactions with a large-scale benchmark showing frequent issues.
Explainable Artificial Intelligence (XAI)has received a great deal of attention recently. Explainability is being presented as a remedy for the distrust of complex and opaque models. Model agnostic methods such as LIME, SHAP, or Break Down promise instance-level interpretability for any complex machine learning model. But how faithful are these additive explanations? Can we rely on additive explanations for non-additive models? In this paper, we (1) examine the behavior of the most popular instance-level explanations under the presence of interactions, (2) introduce a new method that detects interactions for instance-level explanations, (3) perform a large scale benchmark to see how frequently additive explanations may be misleading.