Assessing reliability of explanations in unbalanced datasets: a use-case on the occurrence of frost events
This addresses the challenge of trusting XAI explanations in high-risk applications with unbalanced data, but it is incremental as it builds on existing XAI evaluation techniques.
The study tackled the problem of evaluating the reliability of explanations from XAI methods in unbalanced datasets, particularly for the minority class, and proposed a simple evaluation method using on-manifold neighbor generation and explanation consistency metrics, applied to a frost event dataset.
The usage of eXplainable Artificial Intelligence (XAI) methods has become essential in practical applications, given the increasing deployment of Artificial Intelligence (AI) models and the legislative requirements put forward in the latest years. A fundamental but often underestimated aspect of the explanations is their robustness, a key property that should be satisfied in order to trust the explanations. In this study, we provide some preliminary insights on evaluating the reliability of explanations in the specific case of unbalanced datasets, which are very frequent in high-risk use-cases, but at the same time considerably challenging for both AI models and XAI methods. We propose a simple evaluation focused on the minority class (i.e. the less frequent one) that leverages on-manifold generation of neighbours, explanation aggregation and a metric to test explanation consistency. We present a use-case based on a tabular dataset with numerical features focusing on the occurrence of frost events.