Contrastive Multimodal Fusion with TupleInfoNCE
This work addresses the challenge of effectively fusing multimodal data for representation learning, which is crucial for applications like computer vision and natural language processing, though it appears incremental as it builds on existing contrastive learning frameworks.
The paper tackles the problem of learning multimodal representations by addressing the limitations of existing contrastive methods that either fail to capture complementary synergies or ignore weaker modalities, resulting in significant performance improvements over previous state-of-the-art methods on three downstream tasks.
This paper proposes a method for representation learning of multimodal data using contrastive losses. A traditional approach is to contrast different modalities to learn the information shared between them. However, that approach could fail to learn the complementary synergies between modalities that might be useful for downstream tasks. Another approach is to concatenate all the modalities into a tuple and then contrast positive and negative tuple correspondences. However, that approach could consider only the stronger modalities while ignoring the weaker ones. To address these issues, we propose a novel contrastive learning objective, TupleInfoNCE. It contrasts tuples based not only on positive and negative correspondences but also by composing new negative tuples using modalities describing different scenes. Training with these additional negatives encourages the learning model to examine the correspondences among modalities in the same tuple, ensuring that weak modalities are not ignored. We provide a theoretical justification based on mutual information for why this approach works, and we propose a sample optimization algorithm to generate positive and negative samples to maximize training efficacy. We find that TupleInfoNCE significantly outperforms the previous state of the arts on three different downstream tasks.