Diffusion Visual Counterfactual Explanations
This addresses the need for interpretable AI tools in image classification, particularly for understanding model decisions in complex, multi-class settings like ImageNet, though it is incremental as it builds on existing diffusion and counterfactual explanation techniques.
The paper tackles the problem of generating realistic visual counterfactual explanations for arbitrary ImageNet classifiers, overcoming limitations of prior methods that were restricted to adversarially robust models or produced non-realistic artefacts. The result is a diffusion-based method that produces images with minimal semantic changes but different classifications, achieving high confidence by the classifier.
Visual Counterfactual Explanations (VCEs) are an important tool to understand the decisions of an image classifier. They are 'small' but 'realistic' semantic changes of the image changing the classifier decision. Current approaches for the generation of VCEs are restricted to adversarially robust models and often contain non-realistic artefacts, or are limited to image classification problems with few classes. In this paper, we overcome this by generating Diffusion Visual Counterfactual Explanations (DVCEs) for arbitrary ImageNet classifiers via a diffusion process. Two modifications to the diffusion process are key for our DVCEs: first, an adaptive parameterization, whose hyperparameters generalize across images and models, together with distance regularization and late start of the diffusion process, allow us to generate images with minimal semantic changes to the original ones but different classification. Second, our cone regularization via an adversarially robust model ensures that the diffusion process does not converge to trivial non-semantic changes, but instead produces realistic images of the target class which achieve high confidence by the classifier.