Raw Data Matters: Enhancing Prompt Tuning by Internal Augmentation on Vision-Language Models
This work addresses the cost and inefficiency of external data amplification in prompt tuning for vision-language models, offering a more self-contained approach.
The paper tackles the problem of enhancing prompt tuning for vision-language models by proposing AugPT, which uses internal data augmentation and a gating mechanism to filter noisy samples, resulting in improved performance and generalization without external knowledge.
For CLIP-based prompt tuning, introducing more data as additional knowledge for enhancing fine-tuning process is proved to be an effective approach. Existing data amplification strategies for prompt tuning typically rely on external knowledge (e.g., large language models or pre-structured knowledge bases), resulting in higher costs for data collection and processing, while generally ignoring further utilization of features in image modality. To address this, we propose Augmentation-driven Prompt Tuning (AugPT), a self-contained distillation-based prompt tuning approach using only internal augmentation on raw dataset to better exploit known features. Specifically, AugPT employs self-supervised augmentation on unlabeled images in the training set, and introduces a novel gating mechanism based on consensus test, reusing the pre-trained prompt tuning backbone model to spontaneously filter noisy samples, further enhancing the quality of augmented views. Extensive experiments validate that AugPT simultaneously enhances model performance and generalization capability without using appended external knowledge. The code of AugPT is available at: https://github.com/JREion/AugPT .