MaskMedPaint: Masked Medical Image Inpainting with Diffusion Models for Mitigation of Spurious Correlations
This addresses the issue of biased models in medical settings, where shortcuts can fail across different hospitals, offering a data-driven solution for clinicians to validate modified images, though it is incremental as it adapts existing diffusion models to medical contexts.
The paper tackles the problem of spurious correlations in medical image classification by proposing MaskMedPaint, a method that uses diffusion models to inpaint training images outside key regions, enhancing generalization to new domains with limited unlabeled data, achieving improvements on datasets like ISIC 2018 and Chest X-ray.
Spurious features associated with class labels can lead image classifiers to rely on shortcuts that don't generalize well to new domains. This is especially problematic in medical settings, where biased models fail when applied to different hospitals or systems. In such cases, data-driven methods to reduce spurious correlations are preferred, as clinicians can directly validate the modified images. While Denoising Diffusion Probabilistic Models (Diffusion Models) show promise for natural images, they are impractical for medical use due to the difficulty of describing spurious medical features. To address this, we propose Masked Medical Image Inpainting (MaskMedPaint), which uses text-to-image diffusion models to augment training images by inpainting areas outside key classification regions to match the target domain. We demonstrate that MaskMedPaint enhances generalization to target domains across both natural (Waterbirds, iWildCam) and medical (ISIC 2018, Chest X-ray) datasets, given limited unlabeled target images.