PRISM: High-Resolution & Precise Counterfactual Medical Image Generation using Language-guided Stable Diffusion
This work addresses challenges in developing robust deep learning systems for medical imaging, enabling more reliable classifiers for clinical deployment, though it is incremental in adapting existing vision-language models to a specialized domain.
The authors tackled the problem of generating high-resolution, precise counterfactual medical images to address spurious correlations and data imbalances in medical imaging, achieving unprecedented precision in modifying specific attributes like medical devices and disease features while preserving other characteristics.
Developing reliable and generalizable deep learning systems for medical imaging faces significant obstacles due to spurious correlations, data imbalances, and limited text annotations in datasets. Addressing these challenges requires architectures that are robust to the unique complexities posed by medical imaging data. Rapid advancements in vision-language foundation models within the natural image domain prompt the question of how they can be adapted for medical imaging tasks. In this work, we present PRISM, a framework that leverages foundation models to generate high-resolution, language-guided medical image counterfactuals using Stable Diffusion. Our approach demonstrates unprecedented precision in selectively modifying spurious correlations (the medical devices) and disease features, enabling the removal and addition of specific attributes while preserving other image characteristics. Through extensive evaluation, we show how PRISM advances counterfactual generation and enables the development of more robust downstream classifiers for clinically deployable solutions. To facilitate broader adoption and research, we make our code publicly available at https://github.com/Amarkr1/PRISM.