Sanity checks and improvements for patch visualisation in prototype-based image classification
This work addresses reliability issues in explainable AI for computer vision, showing that current visualizations can mislead users about model behavior, which is incremental but important for improving trust in prototype-based models.
The paper analyzed visualization methods in prototype-based image classification models (ProtoPNet and ProtoTree) and found they incorrectly identify regions of interest, while saliency methods like Smoothgrads or PRP provide more faithful patches, as demonstrated quantitatively using a deletion metric on fine-grained datasets (CUB-200-2011 and Stanford Cars).
In this work, we perform an in-depth analysis of the visualisation methods implemented in two popular self-explaining models for visual classification based on prototypes - ProtoPNet and ProtoTree. Using two fine-grained datasets (CUB-200-2011 and Stanford Cars), we first show that such methods do not correctly identify the regions of interest inside of the images, and therefore do not reflect the model behaviour. Secondly, using a deletion metric, we demonstrate quantitatively that saliency methods such as Smoothgrads or PRP provide more faithful image patches. We also propose a new relevance metric based on the segmentation of the object provided in some datasets (e.g. CUB-200-2011) and show that the imprecise patch visualisations generated by ProtoPNet and ProtoTree can create a false sense of bias that can be mitigated by the use of more faithful methods. Finally, we discuss the implications of our findings for other prototype-based models sharing the same visualisation method.