Fast Diffusion-Based Counterfactuals for Shortcut Removal and Generation
This addresses the problem of spurious correlations in medical imaging for improved model reliability, though it is incremental as it builds on existing diffusion-based techniques.
The paper tackles shortcut learning in medical imaging by proposing a fast diffusion-based method to generate counterfactual images that remove or add shortcut features, enabling detection and quantification of their impact on model predictions. The method achieves significant inference speed-up with comparable image quality to state-of-the-art, as validated on chest X-ray, skin lesion, and CelebA datasets.
Shortcut learning is when a model -- e.g. a cardiac disease classifier -- exploits correlations between the target label and a spurious shortcut feature, e.g. a pacemaker, to predict the target label based on the shortcut rather than real discriminative features. This is common in medical imaging, where treatment and clinical annotations correlate with disease labels, making them easy shortcuts to predict disease. We propose a novel detection and quantification of the impact of potential shortcut features via a fast diffusion-based counterfactual image generation that can synthetically remove or add shortcuts. Via a novel inpainting-based modification we spatially limit the changes made with no extra inference step, encouraging the removal of spatially constrained shortcut features while ensuring that the shortcut-free counterfactuals preserve their remaining image features to a high degree. Using these, we assess how shortcut features influence model predictions. This is enabled by our second contribution: An efficient diffusion-based counterfactual explanation method with significant inference speed-up at comparable image quality as state-of-the-art. We confirm this on two large chest X-ray datasets, a skin lesion dataset, and CelebA. Our code is publicly available at fastdime.compute.dtu.dk.