Vision-Language Model Selection and Reuse for Downstream Adaptation
This addresses the challenge of efficiently adapting VLMs to specific tasks for researchers and practitioners, though it is incremental as it builds on existing VLM selection paradigms.
The paper tackles the problem of selecting the best pre-trained Vision-Language Model (VLM) for downstream tasks by proposing Model Label Learning (MLL), which includes model labeling, selection, and reuse modules, achieving effective results as demonstrated on a benchmark with 49 VLMs and 17 datasets.
Pre-trained Vision-Language Models (VLMs) are becoming increasingly popular across various visual tasks, and several open-sourced VLM variants have been released. However, selecting the best-performing pre-trained VLM for a specific downstream task is challenging since no single VLM can achieve promising performance on all downstream tasks, and evaluating all available VLMs is impossible due to time and data limitations. To address this problem, this paper proposes a novel paradigm to select and reuse VLM for downstream tasks, called Model Label Learning (MLL). The proposal contains three key modules: \emph{model labeling}, which assigns labels to each VLM to describe their specialty and utility; \emph{model selection}, which matches the requirements of the target task with model labels; and \emph{model reuse}, which applies selected VLMs to the target task in an ensemble manner. The proposal is highly computationally efficient and growable since the model labeling process is completed target task independent and the ability could grow with the number of candidate VLMs. We also introduce a new benchmark for evaluating VLM selection methods, including 49 VLMs and 17 target task datasets. Experimental results clearly demonstrate the effectiveness of the proposed method for selecting and reusing VLMs.