When a Relation Tells More Than a Concept: Exploring and Evaluating Classifier Decisions with CoReX
This addresses the need for more specific explanations in complex domains like biology, where pixel-based methods are insufficient, though it is an incremental improvement in explainable AI.
The paper tackles the problem of explaining and evaluating Convolutional Neural Networks (CNNs) by proposing CoReX, a concept- and relation-based explainer that masks relevant concepts and constrains relations in a surrogate model, showing it is faithful to CNN predictions and helps assess classification quality through human evaluation.
Explanations for Convolutional Neural Networks (CNNs) based on relevance of input pixels might be too unspecific to evaluate which and how input features impact model decisions. Especially in complex real-world domains like biology, the presence of specific concepts and of relations between concepts might be discriminating between classes. Pixel relevance is not expressive enough to convey this type of information. In consequence, model evaluation is limited and relevant aspects present in the data and influencing the model decisions might be overlooked. This work presents a novel method to explain and evaluate CNN models, which uses a concept- and relation-based explainer (CoReX). It explains the predictive behavior of a model on a set of images by masking (ir-)relevant concepts from the decision-making process and by constraining relations in a learned interpretable surrogate model. We test our approach with several image data sets and CNN architectures. Results show that CoReX explanations are faithful to the CNN model in terms of predictive outcomes. We further demonstrate through a human evaluation that CoReX is a suitable tool for generating combined explanations that help assessing the classification quality of CNNs. We further show that CoReX supports the identification and re-classification of incorrect or ambiguous classifications.