Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlation
This highlights a critical limitation for practitioners using explanation methods to ensure model reliability in real-world applications, showing incremental insights into their shortcomings.
The study investigated whether post hoc model explanations (feature attribution, concept activation, and training point ranking) are effective for detecting a model's reliance on unknown spurious signals in training data, finding they are ineffective, especially for non-visible artifacts like background blur, and feature attribution methods can erroneously indicate dependence on spurious signals.
We investigate whether three types of post hoc model explanations--feature attribution, concept activation, and training point ranking--are effective for detecting a model's reliance on spurious signals in the training data. Specifically, we consider the scenario where the spurious signal to be detected is unknown, at test-time, to the user of the explanation method. We design an empirical methodology that uses semi-synthetic datasets along with pre-specified spurious artifacts to obtain models that verifiably rely on these spurious training signals. We then provide a suite of metrics that assess an explanation method's reliability for spurious signal detection under various conditions. We find that the post hoc explanation methods tested are ineffective when the spurious artifact is unknown at test-time especially for non-visible artifacts like a background blur. Further, we find that feature attribution methods are susceptible to erroneously indicating dependence on spurious signals even when the model being explained does not rely on spurious artifacts. This finding casts doubt on the utility of these approaches, in the hands of a practitioner, for detecting a model's reliance on spurious signals.