Multi-modal contrastive learning adapts to intrinsic dimensions of shared latent variables
This work provides theoretical insights into a widely used self-supervised learning technique, which is incremental as it builds on existing methods like CLIP.
The paper tackles the problem of understanding the theoretical properties of multi-modal contrastive learning beyond linear representations and specific data distributions, revealing that it adapts to intrinsic data dimensions, which can be lower than user-specified dimensions, as demonstrated in experiments on synthetic and real-world datasets.
Multi-modal contrastive learning as a self-supervised representation learning technique has achieved great success in foundation model training, such as CLIP~\citep{radford2021learning}. In this paper, we study the theoretical properties of the learned representations from multi-modal contrastive learning beyond linear representations and specific data distributions. Our analysis reveals that, enabled by temperature optimization, multi-modal contrastive learning not only maximizes mutual information between modalities but also adapts to intrinsic dimensions of data, which can be much lower than user-specified dimensions for representation vectors. Experiments on both synthetic and real-world datasets demonstrate the ability of contrastive learning to learn low-dimensional and informative representations, bridging theoretical insights and practical performance.