Closing the gap in multimodal medical representation alignment
This work addresses a critical bottleneck in medical AI by enhancing alignment between radiology images and clinical text, though it is incremental as it builds on existing CLIP methods.
The paper tackled the modality gap problem in multimodal medical representation alignment, where CLIP-based contrastive losses cause sparse and fragmented latent spaces, and proposed a modality-agnostic framework that improved cross-modal retrieval and image captioning for radiology images and clinical text.
In multimodal learning, CLIP has emerged as the de-facto approach for mapping different modalities into a shared latent space by bringing semantically similar representations closer while pushing apart dissimilar ones. However, CLIP-based contrastive losses exhibit unintended behaviors that negatively impact true semantic alignment, leading to sparse and fragmented latent spaces. This phenomenon, known as the modality gap, has been partially mitigated for standard text and image pairs but remains unknown and unresolved in more complex multimodal settings, such as the medical domain. In this work, we study this phenomenon in the latter case, revealing that the modality gap is present also in medical alignment, and we propose a modality-agnostic framework that closes this gap, ensuring that semantically related representations are more aligned, regardless of their source modality. Our method enhances alignment between radiology images and clinical text, improving cross-modal retrieval and image captioning.