Integrated Structural Prompt Learning for Vision-Language Models
This addresses a specific challenge in fine-tuning vision-language models for downstream tasks, representing an incremental improvement over existing prompt learning methods.
The paper tackled the problem of ignoring structural relationships between learnable prompts and tokens in vision-language models, and balancing base and new class performance, by proposing an Integrated Structural Prompt (ISP) with self-structural and cross-structural modules and a sample probing module, achieving competitive performance in base-to-new generalization, cross-dataset evaluation, and domain generalization.
Prompt learning methods have significantly extended the transferability of pre-trained Vision-Language Models (VLMs) like CLIP for various downstream tasks. These methods adopt handcraft templates or learnable vectors to provide text or image instructions in fine-tuning VLMs. However, most existing works ignore the structural relationships between learnable prompts and tokens within and between modalities. Moreover, balancing the performance of base and new classes remains a significant challenge. In this paper, we propose an Integrated Structural Prompt (ISP) for VLMs to enhance the interaction of information representations between the text and image branches. ISP introduces self-structural and cross-structural prompt modules to model the structural relationships between learnable prompts and frozen tokens within and across modalities. This enables efficient information transfer while preserving feature stability. Additionally, we propose a sample probing module that dynamically adjusts loss coefficients based on sample difficulty, preventing the mode from overfitting to simple samples and improving generalization ability to new classes. Extensive experiments on three widely used settings: base-to-new generalization, cross-dataset evaluation, and domain generalization demonstrate that the proposed ISP achieves competitive performance against state-of-the-art methods.