Weak Distribution Detectors Lead to Stronger Generalizability of Vision-Language Prompt Tuning
This work addresses generalization challenges in vision-language models for few-shot learning, offering a test-time method to boost existing techniques without retraining, though it is incremental as it builds on prior prompt tuning approaches.
The paper tackles the problem of improving generalization in vision-language models during few-shot fine-tuning by using out-of-distribution detection to dynamically fuse zero-shot and few-shot classifiers, resulting in increased harmonic mean scores by 2.6 and 1.5 percentage points for CoOp and ProGrad across 11 datasets.
We propose a generalized method for boosting the generalization ability of pre-trained vision-language models (VLMs) while fine-tuning on downstream few-shot tasks. The idea is realized by exploiting out-of-distribution (OOD) detection to predict whether a sample belongs to a base distribution or a novel distribution and then using the score generated by a dedicated competition based scoring function to fuse the zero-shot and few-shot classifier. The fused classifier is dynamic, which will bias towards the zero-shot classifier if a sample is more likely from the distribution pre-trained on, leading to improved base-to-novel generalization ability. Our method is performed only in test stage, which is applicable to boost existing methods without time-consuming re-training. Extensive experiments show that even weak distribution detectors can still improve VLMs' generalization ability. Specifically, with the help of OOD detectors, the harmonic mean of CoOp and ProGrad increase by 2.6 and 1.5 percentage points over 11 recognition datasets in the base-to-novel setting.