A Statistical Theory of Contrastive Pre-training and Multimodal Generative AI
This provides foundational theoretical insights for multimodal generative AI systems, addressing a key gap in understanding why contrastive pre-training works.
The paper tackles the lack of theoretical understanding of contrastive pre-training in multimodal AI by developing a framework showing that near-minimizers of the contrastive loss are approximately sufficient statistics, enabling adaptability to downstream tasks like zero-shot classification and vision-language models, with numerical simulations validating strong generalization performance.
Multi-modal generative AI systems, such as those combining vision and language, rely on contrastive pre-training to learn representations across different modalities. While their practical benefits are widely acknowledged, a rigorous theoretical understanding of the contrastive pre-training framework remains limited. This paper develops a theoretical framework to explain the success of contrastive pre-training in downstream tasks, such as zero-shot classification, conditional diffusion models, and vision-language models. We introduce the concept of approximate sufficient statistics, a generalization of the classical sufficient statistics, and show that near-minimizers of the contrastive pre-training loss are approximately sufficient, making them adaptable to diverse downstream tasks. We further propose the Joint Generative Hierarchical Model for the joint distribution of images and text, showing that transformers can efficiently approximate relevant functions within this model via belief propagation. Building on this framework, we derive sample complexity guarantees for multi-modal learning based on contrastive pre-trained representations. Numerical simulations validate these theoretical findings, demonstrating the strong generalization performance of contrastively pre-trained transformers in various multi-modal tasks.