Mitigate the Gap: Investigating Approaches for Improving Cross-Modal Alignment in CLIP
This work addresses the modality gap problem in CLIP for researchers and practitioners in vision-language tasks, but it is incremental as it builds on existing CLIP methods.
The paper tackled the modality gap in CLIP's embedding space, which causes sparsity and disconnection, by proposing AlignCLIP to investigate parameter sharing and intra-modality separation, resulting in noticeable enhancements in cross-modal alignment and improved performance in zero-shot and fine-tuning evaluations.
Contrastive Language--Image Pre-training (CLIP) has manifested remarkable improvements in zero-shot classification and cross-modal vision-language tasks. Yet, from a geometrical point of view, the CLIP embedding space has been found to have a pronounced modality gap. This gap renders the embedding space overly sparse and disconnected, with different modalities being densely distributed in distinct subregions of the hypersphere. In this work, we aim at answering three main questions: 1. Does sharing the parameter space between the multi-modal encoders reduce the modality gap? 2. Can the gap be mitigated by pushing apart the uni-modal embeddings via intra-modality separation? 3. How do these gap reduction approaches affect the downstream performance? We design AlignCLIP, in order to answer these questions and through extensive experiments, we show that AlignCLIP achieves noticeable enhancements in the cross-modal alignment of the embeddings, and thereby, reduces the modality gap, while improving the performance across several zero-shot and fine-tuning downstream evaluations.