COBRA: Contrastive Bi-Modal Representation Algorithm
This addresses the challenge of aligning multi-modal data for applications like cross-modal retrieval, offering a novel method that improves performance in downstream tasks.
The paper tackles the problem of modality gaps in multi-modal data by introducing COBRA, a framework that trains image and text embeddings jointly using contrastive learning, resulting in significant reduction of the modality gap and outperforming existing work on four downstream tasks across seven benchmark datasets.
There are a wide range of applications that involve multi-modal data, such as cross-modal retrieval, visual question-answering, and image captioning. Such applications are primarily dependent on aligned distributions of the different constituent modalities. Existing approaches generate latent embeddings for each modality in a joint fashion by representing them in a common manifold. However these joint embedding spaces fail to sufficiently reduce the modality gap, which affects the performance in downstream tasks. We hypothesize that these embeddings retain the intra-class relationships but are unable to preserve the inter-class dynamics. In this paper, we present a novel framework COBRA that aims to train two modalities (image and text) in a joint fashion inspired by the Contrastive Predictive Coding (CPC) and Noise Contrastive Estimation (NCE) paradigms which preserve both inter and intra-class relationships. We empirically show that this framework reduces the modality gap significantly and generates a robust and task agnostic joint-embedding space. We outperform existing work on four diverse downstream tasks spanning across seven benchmark cross-modal datasets.