Bayesian Cross-Modal Alignment Learning for Few-Shot Out-of-Distribution Generalization
This addresses the challenge of improving generalization for few-shot learning in out-of-distribution scenarios, which is critical for real-world applications where data is scarce and distributions shift, though it appears incremental as it builds on existing cross-modal and Bayesian approaches.
The paper tackles the problem of few-shot out-of-distribution generalization in deep neural networks, where models overfit on limited data and struggle with correlation and diversity shifts, and proposes a Bayesian cross-modal alignment learning method that achieves state-of-the-art performance on two-dimensional distribution shifts, with more stable generalization on unseen classes compared to CLIP-like models.
Recent advances in large pre-trained models showed promising results in few-shot learning. However, their generalization ability on two-dimensional Out-of-Distribution (OoD) data, i.e., correlation shift and diversity shift, has not been thoroughly investigated. Researches have shown that even with a significant amount of training data, few methods can achieve better performance than the standard empirical risk minimization method (ERM) in OoD generalization. This few-shot OoD generalization dilemma emerges as a challenging direction in deep neural network generalization research, where the performance suffers from overfitting on few-shot examples and OoD generalization errors. In this paper, leveraging a broader supervision source, we explore a novel Bayesian cross-modal image-text alignment learning method (Bayes-CAL) to address this issue. Specifically, the model is designed as only text representations are fine-tuned via a Bayesian modelling approach with gradient orthogonalization loss and invariant risk minimization (IRM) loss. The Bayesian approach is essentially introduced to avoid overfitting the base classes observed during training and improve generalization to broader unseen classes. The dedicated loss is introduced to achieve better image-text alignment by disentangling the causal and non-casual parts of image features. Numerical experiments demonstrate that Bayes-CAL achieved state-of-the-art OoD generalization performances on two-dimensional distribution shifts. Moreover, compared with CLIP-like models, Bayes-CAL yields more stable generalization performances on unseen classes. Our code is available at https://github.com/LinLLLL/BayesCAL.