Counterfactual Evaluation for Explainable AI
This addresses the challenge of ensuring that explanations accurately represent model reasoning, which is crucial for users relying on explainable AI, though it is an incremental improvement over existing evaluation methods.
The paper tackles the problem of evaluating the faithfulness of explanations in machine learning by proposing a counterfactual reasoning methodology, which achieves top correlation with ground truth on several datasets.
While recent years have witnessed the emergence of various explainable methods in machine learning, to what degree the explanations really represent the reasoning process behind the model prediction -- namely, the faithfulness of explanation -- is still an open problem. One commonly used way to measure faithfulness is \textit{erasure-based} criteria. Though conceptually simple, erasure-based criterion could inevitably introduce biases and artifacts. We propose a new methodology to evaluate the faithfulness of explanations from the \textit{counterfactual reasoning} perspective: the model should produce substantially different outputs for the original input and its corresponding counterfactual edited on a faithful feature. Specially, we introduce two algorithms to find the proper counterfactuals in both discrete and continuous scenarios and then use the acquired counterfactuals to measure faithfulness. Empirical results on several datasets show that compared with existing metrics, our proposed counterfactual evaluation method can achieve top correlation with the ground truth under diffe