On the Generalization of Multi-modal Contrastive Learning
This work provides theoretical insights into MMCL for researchers in multi-modal learning, though it is incremental as it builds on existing contrastive learning frameworks.
The paper tackles the lack of theoretical understanding of multi-modal contrastive learning (MMCL) by connecting it to asymmetric matrix factorization, establishing generalization guarantees and showing that MMCL implicitly performs self-supervised contrastive learning with text-induced positive pairs, which provably benefits downstream generalization. It proposes CLIP-guided resampling methods that significantly improve SSCL performance on ImageNet by leveraging multi-modal information.
Multi-modal contrastive learning (MMCL) has recently garnered considerable interest due to its superior performance in visual tasks, achieved by embedding multi-modal data, such as visual-language pairs. However, there still lack theoretical understandings of how MMCL extracts useful visual representation from multi-modal pairs, and particularly, how MMCL outperforms previous approaches like self-supervised contrastive learning (SSCL). In this paper, by drawing an intrinsic connection between MMCL and asymmetric matrix factorization, we establish the first generalization guarantees of MMCL for visual downstream tasks. Based on this framework, we further unify MMCL and SSCL by showing that MMCL implicitly performs SSCL with (pseudo) positive pairs induced by text pairs. Through this unified perspective, we characterize the advantage of MMCL by showing that text pairs induce more semantically consistent and diverse positive pairs, which, according to our analysis, provably benefit downstream generalization. Inspired by this finding, we propose CLIP-guided resampling methods to significantly improve the downstream performance of SSCL on ImageNet by leveraging multi-modal information. Code is available at https://github.com/PKU-ML/CLIP-Help-SimCLR.