Generalized Contrastive Learning for Universal Multimodal Retrieval
This addresses a critical challenge in multimodal retrieval for AI systems handling diverse data types, though it is incremental as it builds on existing models like CLIP.
The paper tackles the problem of degraded performance in cross-modal retrieval models when retrieving keys composed of fused image-text modalities, proposing Generalized Contrastive Learning (GCL) to improve multimodal retrieval without new dataset curation, showing consistent performance improvements on benchmarks like M-BEIR, MMEB, and CoVR.
Despite their consistent performance improvements, cross-modal retrieval models (e.g., CLIP) show degraded performances with retrieving keys composed of fused image-text modality (e.g., Wikipedia pages with both images and text). To address this critical challenge, multimodal retrieval has been recently explored to develop a unified single retrieval model capable of retrieving keys across diverse modality combinations. A common approach involves constructing new composed sets of image-text triplets (e.g., retrieving a pair of image and text given a query image). However, such an approach requires careful curation to ensure the dataset quality and fails to generalize to unseen modality combinations. To overcome these limitations, this paper proposes Generalized Contrastive Learning (GCL), a novel loss formulation that improves multimodal retrieval performance without the burdensome need for new dataset curation. Specifically, GCL operates by enforcing contrastive learning across all modalities within a mini-batch, utilizing existing image-caption paired datasets to learn a unified representation space. We demonstrate the effectiveness of GCL by showing consistent performance improvements on off-the-shelf multimodal retrieval models (e.g., VISTA, CLIP, and TinyCLIP) using the M-BEIR, MMEB, and CoVR benchmarks.