Explaining and Mitigating the Modality Gap in Contrastive Multimodal Learning
This work addresses a fundamental issue in multimodal learning for AI researchers, offering incremental insights into model behavior and practical mitigation strategies.
The paper tackled the modality gap in contrastive multimodal learning by analyzing its emergence through gradient flow dynamics and identifying mismatched data pairs and a learnable temperature as key causes, with experiments showing that mitigating this gap improves performance in tasks like image-text retrieval.
Multimodal learning has recently gained significant popularity, demonstrating impressive performance across various zero-shot classification tasks and a range of perceptive and generative applications. Models such as Contrastive Language-Image Pretraining (CLIP) are designed to bridge different modalities, such as images and text, by learning a shared representation space through contrastive learning. Despite their success, the working mechanisms underlying multimodal learning are not yet well understood. Notably, these models often exhibit a modality gap, where different modalities occupy distinct regions within the shared representation space. In this work, we conduct an in-depth analysis of the emergence of modality gap by characterizing the gradient flow learning dynamics. Specifically, we identify the critical roles of mismatched data pairs and a learnable temperature parameter in causing and perpetuating the modality gap during training. Furthermore, our theoretical insights are validated through experiments on practical CLIP models. These findings provide principled guidance for mitigating the modality gap, including strategies such as appropriate temperature scheduling and modality swapping. Additionally, we demonstrate that closing the modality gap leads to improved performance on tasks such as image-text retrieval.