Closing the Modality Gap for Mixed Modality Search
This addresses the problem of intra-modal bias and fusion failure in mixed modality search for real-world applications, representing a strong incremental improvement.
The paper tackled the modality gap in mixed modality search by analyzing CLIP's limitations and proposing GR-CLIP, a lightweight calibration method that improved NDCG@10 by up to 26 percentage points over CLIP and outperformed other models with 75x less compute.
Mixed modality search -- retrieving information across a heterogeneous corpus composed of images, texts, and multimodal documents -- is an important yet underexplored real-world application. In this work, we investigate how contrastive vision-language models, such as CLIP, perform on the mixed modality search task. Our analysis reveals a critical limitation: these models exhibit a pronounced modality gap in the embedding space, where image and text embeddings form distinct clusters, leading to intra-modal ranking bias and inter-modal fusion failure. To address this issue, we propose GR-CLIP, a lightweight post-hoc calibration method that removes the modality gap in CLIP's embedding space. Evaluated on MixBench -- the first benchmark specifically designed for mixed modality search -- GR-CLIP improves NDCG@10 by up to 26 percentage points over CLIP, surpasses recent vision-language generative embedding models by 4 percentage points, while using 75x less compute.