CP-Prompt: Composition-Based Cross-modal Prompting for Domain-Incremental Continual Learning
This addresses the challenge of enabling models to continuously learn from novel data with different feature distributions without forgetting old ones, which is incremental as it builds on existing prompting strategies.
The paper tackles the problem of high forgetting rates in cross-modal domain-incremental learning by proposing CP-Prompt, a framework that trains limited parameters to instruct a pre-trained model to learn new domains without forgetting old ones, achieving superiority over state-of-the-art baselines on three widely evaluated tasks.
The key challenge of cross-modal domain-incremental learning (DIL) is to enable the learning model to continuously learn from novel data with different feature distributions under the same task without forgetting old ones. However, existing top-performing methods still cause high forgetting rates, by lacking intra-domain knowledge extraction and inter-domain common prompting strategy. In this paper, we propose a simple yet effective framework, CP-Prompt, by training limited parameters to instruct a pre-trained model to learn new domains and avoid forgetting existing feature distributions. CP-Prompt captures intra-domain knowledge by compositionally inserting personalized prompts on multi-head self-attention layers and then learns the inter-domain knowledge with a common prompting strategy. CP-Prompt shows superiority compared with state-of-the-art baselines among three widely evaluated DIL tasks. The source code is available at https://github.com/dannis97500/CP_Prompt.