Understanding the Robustness of Multi-modal Contrastive Learning to Distribution Shift
This provides theoretical insights into a widely used AI method, addressing robustness for researchers and practitioners, though it is incremental in explaining existing empirical success.
The paper tackled the problem of understanding why multimodal contrastive learning (MMCL) like CLIP is robust to distribution shifts, uncovering two mechanisms—intra-class contrasting and inter-class feature sharing—that prevent spurious features from overshadowing core ones, leading to superior zero-shot classification accuracy under shift.
Recently, multimodal contrastive learning (MMCL) approaches, such as CLIP, have achieved a remarkable success in learning representations that are robust against distribution shift and generalize to new domains. Despite the empirical success, the mechanism behind learning such generalizable representations is not understood. In this work, we rigorously analyze this problem and uncover two mechanisms behind MMCL's robustness: \emph{intra-class contrasting}, which allows the model to learn features with a high variance, and \emph{inter-class feature sharing}, where annotated details in one class help learning other classes better. Both mechanisms prevent spurious features that are over-represented in the training data to overshadow the generalizable core features. This yields superior zero-shot classification accuracy under distribution shift. Furthermore, we theoretically demonstrate the benefits of using rich captions on robustness and explore the effect of annotating different types of details in the captions. We validate our theoretical findings through experiments, including a well-designed synthetic experiment and an experiment involving training CLIP models on MSCOCO/Conceptual Captions and evaluating them on shifted ImageNets.