Exploring SAIG Methods for an Objective Evaluation of XAI
This work addresses the problem of objective evaluation for XAI researchers and practitioners, but it is incremental as it focuses on reviewing and analyzing existing SAIG methods rather than proposing a new solution.
The paper tackles the challenge of objectively evaluating eXplainable Artificial Intelligence (XAI) methods due to the lack of a universally correct ground truth, by reviewing and analyzing Synthetic Artificial Intelligence Ground truth (SAIG) methods that generate artificial ground truths for direct evaluation. It introduces a novel taxonomy classifying these approaches and finds a concerning lack of consensus on effective XAI evaluation techniques, highlighting the need for further research and standardization.
The evaluation of eXplainable Artificial Intelligence (XAI) methods is a rapidly growing field, characterized by a wide variety of approaches. This diversity highlights the complexity of the XAI evaluation, which, unlike traditional AI assessment, lacks a universally correct ground truth for the explanation, making objective evaluation challenging. One promising direction to address this issue involves the use of what we term Synthetic Artificial Intelligence Ground truth (SAIG) methods, which generate artificial ground truths to enable the direct evaluation of XAI techniques. This paper presents the first review and analysis of SAIG methods. We introduce a novel taxonomy to classify these approaches, identifying seven key features that distinguish different SAIG methods. Our comparative study reveals a concerning lack of consensus on the most effective XAI evaluation techniques, underscoring the need for further research and standardization in this area.