Evaluating the Correctness of Explainable AI Algorithms for Classification
This work addresses the lack of ground truth for validating feature attribution methods in XAI, which is a problem for researchers and practitioners needing reliable explanations, though it is incremental as it builds on existing explainers and focuses on binary classification.
The authors tackled the problem of evaluating the correctness of explainable AI algorithms for classification by creating datasets with known explanation ground truth, showing that SHAP provides more accurate explanations than LIME and that explanation accuracy correlates with classification accuracy and dataset complexity.
Explainable AI has attracted much research attention in recent years with feature attribution algorithms, which compute "feature importance" in predictions, becoming increasingly popular. However, there is little analysis of the validity of these algorithms as there is no "ground truth" in the existing datasets to validate their correctness. In this work, we develop a method to quantitatively evaluate the correctness of XAI algorithms by creating datasets with known explanation ground truth. To this end, we focus on the binary classification problems. String datasets are constructed using formal language derived from a grammar. A string is positive if and only if a certain property is fulfilled. Symbols serving as explanation ground truth in a positive string are part of an explanation if and only if they contributes to fulfilling the property. Two popular feature attribution explainers, Local Interpretable Model-agnostic Explanations (LIME) and SHapley Additive exPlanations (SHAP), are used in our experiments.We show that: (1) classification accuracy is positively correlated with explanation accuracy; (2) SHAP provides more accurate explanations than LIME; (3) explanation accuracy is negatively correlated with dataset complexity.