Leveraging Vision-Language Foundation Models to Reveal Hidden Image-Attribute Relationships in Medical Imaging
This work addresses the challenge of uncovering hidden image-attribute relationships in medical imaging, which could aid in diagnosis and dataset analysis, though it is incremental as it builds on existing foundation models.
The paper tackled the problem of identifying hidden data properties in medical imaging by fine-tuning vision-language foundation models, showing they can generate high-resolution, precisely edited chest x-ray images compared to structural causal models and reveal previously obscured relationships.
Vision-language foundation models (VLMs) have shown impressive performance in guiding image generation through text, with emerging applications in medical imaging. In this work, we are the first to investigate the question: 'Can fine-tuned foundation models help identify critical, and possibly unknown, data properties?' By evaluating our proposed method on a chest x-ray dataset, we show that these models can generate high-resolution, precisely edited images compared to methods that rely on Structural Causal Models (SCMs) according to numerous metrics. For the first time, we demonstrate that fine-tuned VLMs can reveal hidden data relationships that were previously obscured due to available metadata granularity and model capacity limitations. Our experiments demonstrate both the potential of these models to reveal underlying dataset properties while also exposing the limitations of fine-tuned VLMs for accurate image editing and susceptibility to biases and spurious correlations.