Any-Shift Prompting for Generalization over Distributions
This addresses the generalization issue in image-language models for vision tasks, but it is incremental as it builds on existing prompt learning frameworks.
The paper tackles the problem of conventional prompt learning methods overfitting to training distributions and losing generalization on test distributions, proposing any-shift prompting to improve generalization across distribution shifts, with experiments on twenty-three datasets showing effectiveness.
Image-language models with prompt learning have shown remarkable advances in numerous downstream vision tasks. Nevertheless, conventional prompt learning methods overfit their training distribution and lose the generalization ability on test distributions. To improve generalization across various distribution shifts, we propose any-shift prompting: a general probabilistic inference framework that considers the relationship between training and test distributions during prompt learning. We explicitly connect training and test distributions in the latent space by constructing training and test prompts in a hierarchical architecture. Within this framework, the test prompt exploits the distribution relationships to guide the generalization of the CLIP image-language model from training to any test distribution. To effectively encode the distribution information and their relationships, we further introduce a transformer inference network with a pseudo-shift training mechanism. The network generates the tailored test prompt with both training and test information in a feedforward pass, avoiding extra training costs at test time. Extensive experiments on twenty-three datasets demonstrate the effectiveness of any-shift prompting on the generalization over various distribution shifts.