A Mathematical Perspective On Contrastive Learning
This work provides a theoretical framework for contrastive learning that could enhance multimodal algorithms like crossmodal retrieval and generative models, though it appears incremental as it builds on existing methods with specific mathematical refinements.
The paper tackles the problem of interpreting multimodal contrastive learning from a probabilistic perspective, framing it as optimizing encoders to define conditional distributions, and introduces generalizations like novel loss functions and alignment metrics, resulting in new algorithm variants for tasks such as mode-seeking and generation, with experimental validation on datasets including MNIST and an oceanography application.
Multimodal contrastive learning is a methodology for linking different data modalities; the canonical example is linking image and text data. The methodology is typically framed as the identification of a set of encoders, one for each modality, that align representations within a common latent space. In this work, we focus on the bimodal setting and interpret contrastive learning as the optimization of (parameterized) encoders that define conditional probability distributions, for each modality conditioned on the other, consistent with the available data. This provides a framework for multimodal algorithms such as crossmodal retrieval, which identifies the mode of one of these conditional distributions, and crossmodal classification, which is similar to retrieval but includes a fine-tuning step to make it task specific. The framework we adopt also gives rise to crossmodal generative models. This probabilistic perspective suggests two natural generalizations of contrastive learning: the introduction of novel probabilistic loss functions, and the use of alternative metrics for measuring alignment in the common latent space. We study these generalizations of the classical approach in the multivariate Gaussian setting. In this context we view the latent space identification as a low-rank matrix approximation problem. This allows us to characterize the capabilities of loss functions and alignment metrics to approximate natural statistics, such as conditional means and covariances; doing so yields novel variants on contrastive learning algorithms for specific mode-seeking and for generative tasks. The framework we introduce is also studied through numerical experiments on multivariate Gaussians, the labeled MNIST dataset, and on a data assimilation application arising in oceanography.