Bayesian Test-Time Adaptation for Vision-Language Models
This addresses the challenge of test-time adaptation for vision-language models in real-world applications, though it is incremental as it builds on existing methods by incorporating prior updates.
The paper tackles the problem of adapting vision-language models like CLIP to out-of-distribution test data by proposing Bayesian Class Adaptation (BCA), which updates both class embeddings and priors based on Bayes theorem, resulting in higher prediction accuracy and improved efficiency in inference rates and memory usage.
Test-time adaptation with pre-trained vision-language models, such as CLIP, aims to adapt the model to new, potentially out-of-distribution test data. Existing methods calculate the similarity between visual embedding and learnable class embeddings, which are initialized by text embeddings, for zero-shot image classification. In this work, we first analyze this process based on Bayes theorem, and observe that the core factors influencing the final prediction are the likelihood and the prior. However, existing methods essentially focus on adapting class embeddings to adapt likelihood, but they often ignore the importance of prior. To address this gap, we propose a novel approach, \textbf{B}ayesian \textbf{C}lass \textbf{A}daptation (BCA), which in addition to continuously updating class embeddings to adapt likelihood, also uses the posterior of incoming samples to continuously update the prior for each class embedding. This dual updating mechanism allows the model to better adapt to distribution shifts and achieve higher prediction accuracy. Our method not only surpasses existing approaches in terms of performance metrics but also maintains superior inference rates and memory usage, making it highly efficient and practical for real-world applications.