Enhancing Continual Learning of Vision-Language Models via Dynamic Prefix Weighting
For researchers working on continual learning of vision-language models, this work addresses a known limitation of prefix-tuning methods by introducing dynamic weighting, leading to improved performance.
The paper tackles domain-class incremental learning for vision-language models, where previous prefix-tuning methods uniformly weight prefixes. The proposed Dynamic Prefix Weighting (DPW) achieves state-of-the-art performance by dynamically assigning weights to prefixes and adapters based on token importance.
We investigate recently introduced domain-class incremental learning scenarios for vision-language models (VLMs). Recent works address this challenge using parameter-efficient methods, such as prefix-tuning or adapters, which facilitate model adaptation to downstream tasks by incorporating task-specific information into input tokens through additive vectors. However, previous approaches often normalize the weights of these vectors, disregarding the fact that different input tokens require different degrees of adjustment. To overcome this issue, we propose Dynamic Prefix Weighting (DPW), a framework that dynamically assigns weights to prefixes, complemented by adapters. DPW consists of 1) a gating module that adjusts the weights of each prefix based on the importance of the corresponding input token, and 2) a weighting mechanism that derives adapter output weights as a residual of prefix-tuning weights, ensuring that adapters are utilized only when necessary. Experimental results demonstrate that our method achieves state-of-the-art performance in domain-class incremental learning scenarios for VLMs. The code is available at: https://github.com/YonseiML/dpw.