Do not explain without context: addressing the blind spot of model explanations
This addresses a critical blind spot for analysts, auditors, and stakeholders in monitoring and auditing machine learning models, highlighting an incremental but important oversight in XAI practices.
The paper tackles the problem that many model explanation methods, such as Shapley values, are sensitive to the choice of reference data distribution, showing that small changes can drastically alter explanations, like reversing trends or conclusions.
The increasing number of regulations and expectations of predictive machine learning models, such as so called right to explanation, has led to a large number of methods promising greater interpretability. High demand has led to a widespread adoption of XAI techniques like Shapley values, Partial Dependence profiles or permutational variable importance. However, we still do not know enough about their properties and how they manifest in the context in which explanations are created by analysts, reviewed by auditors, and interpreted by various stakeholders. This paper highlights a blind spot which, although critical, is often overlooked when monitoring and auditing machine learning models: the effect of the reference data on the explanation calculation. We discuss that many model explanations depend directly or indirectly on the choice of the referenced data distribution. We showcase examples where small changes in the distribution lead to drastic changes in the explanations, such as a change in trend or, alarmingly, a conclusion. Consequently, we postulate that obtaining robust and useful explanations always requires supporting them with a broader context.