Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities
This addresses the need for flexible multi-modal learning in domains like robotics and healthcare, offering a novel method for capturing joint information across unlimited modalities.
The paper tackled the problem of learning joint representations across more than two modalities, showing that pairwise CLIP fails to capture higher-order information, and introduced Symile, which outperformed CLIP on cross-modal tasks, achieving improvements on datasets including 33M multilingual samples and clinical data.
Contrastive learning methods, such as CLIP, leverage naturally paired data-for example, images and their corresponding text captions-to learn general representations that transfer efficiently to downstream tasks. While such approaches are generally applied to two modalities, domains such as robotics, healthcare, and video need to support many types of data at once. We show that the pairwise application of CLIP fails to capture joint information between modalities, thereby limiting the quality of the learned representations. To address this issue, we present Symile, a simple contrastive learning approach that captures higher-order information between any number of modalities. Symile provides a flexible, architecture-agnostic objective for learning modality-specific representations. To develop Symile's objective, we derive a lower bound on total correlation, and show that Symile representations for any set of modalities form a sufficient statistic for predicting the remaining modalities. Symile outperforms pairwise CLIP, even with modalities missing in the data, on cross-modal classification and retrieval across several experiments including on an original multilingual dataset of 33M image, text and audio samples and a clinical dataset of chest X-rays, electrocardiograms, and laboratory measurements. All datasets and code used in this work are publicly available at https://github.com/rajesh-lab/symile.