Feature Projection Learning for Better Vision-Language Reasoning
This addresses the challenge of adapting CLIP for downstream tasks with limited supervision, offering a more efficient and effective solution, though it appears incremental as it builds on existing adaptation methods.
The paper tackles the problem of efficiently adapting vision-language pre-trained models like CLIP to downstream tasks by proposing Feature Projection Learning (FPL), which projects class prototypes into image feature space and reconstructs features, achieving superior accuracy over state-of-the-art methods.
Vision-Language Pre-Trained models, notably CLIP, that utilize contrastive learning have proven highly adept at extracting generalizable visual features. To inherit the well-learned knowledge of VLP models for downstream tasks, several approaches aim to adapt them efficiently with limited supervision. However, these methods either suffer from limited performance, excessive learnable parameters, or extended training times, all of which hinder their effectiveness in adapting the CLIP model to downstream tasks. In this work, we propose a simple yet efficient and effective method called \textit{\textbf{F}eature \textbf{P}rojection \textbf{L}earning(FPL)} to address these problems. Specifically, we develop a projection model that projects class prototype features into the query image feature space and reconstructs the query image feature map. The negative average squared reconstruction error is used as the class score. In this way, we transform the classification problem into a feature projection problem. The final output of this method is a combination of the prediction from the projection model and the original pre-trained CLIP. Comprehensive empirical evaluations confirm that FPL delivers superior accuracy, surpassing the current state-of-the-art methods by a substantial margin.