A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation
This work addresses the need for efficient adaptation of CLIP for resource-constrained devices, offering an incremental improvement by revisiting a classical algorithm.
The paper tackles the problem of reducing computational costs for adapting CLIP to downstream tasks by proposing a training-free method using Gaussian Discriminant Analysis (GDA) and ensembling with zero-shot classification, achieving comparable or superior results to state-of-the-art methods on 17 datasets across few-shot, imbalanced, and out-of-distribution settings.
Contrastive Language-Image Pretraining (CLIP) has gained popularity for its remarkable zero-shot capacity. Recent research has focused on developing efficient fine-tuning methods, such as prompt learning and adapter, to enhance CLIP's performance in downstream tasks. However, these methods still require additional training time and computational resources, which is undesirable for devices with limited resources. In this paper, we revisit a classical algorithm, Gaussian Discriminant Analysis (GDA), and apply it to the downstream classification of CLIP. Typically, GDA assumes that features of each class follow Gaussian distributions with identical covariance. By leveraging Bayes' formula, the classifier can be expressed in terms of the class means and covariance, which can be estimated from the data without the need for training. To integrate knowledge from both visual and textual modalities, we ensemble it with the original zero-shot classifier within CLIP. Extensive results on 17 datasets validate that our method surpasses or achieves comparable results with state-of-the-art methods on few-shot classification, imbalanced learning, and out-of-distribution generalization. In addition, we extend our method to base-to-new generalization and unsupervised learning, once again demonstrating its superiority over competing approaches. Our code is publicly available at \url{https://github.com/mrflogs/ICLR24}.