Efficient Prompting for Continual Adaptation to Missing Modalities
This addresses the issue of missing modalities in real-world applications like equipment failures or privacy concerns, but it is incremental as it builds on existing prompting and continual learning techniques.
The paper tackles the problem of performance degradation in pre-trained models when adapting to downstream datasets with missing modalities in continual learning settings, and introduces a prompting method that outperforms state-of-the-art approaches on three public datasets.
Missing modality issues are common in real-world applications, arising from factors such as equipment failures and privacy concerns. When fine-tuning pre-trained models on downstream datasets with missing modalities, performance can degrade significantly. Current methods often aggregate various missing cases to train recovery modules or align multimodal features, resulting in suboptimal performance, high computational costs, and the risk of catastrophic forgetting in continual environments where data arrives sequentially. In this paper, we formulate the dynamic missing modality problem as a continual learning task and introduce the continual multimodal missing modality task. To address this challenge efficiently, we introduce three types of prompts: modality-specific, task-aware, and task-specific prompts. These prompts enable the model to learn intra-modality, inter-modality, intra-task, and inter-task features. Furthermore, we propose a contrastive task interaction strategy to explicitly learn prompts correlating different modalities. We conduct extensive experiments on three public datasets, where our method consistently outperforms state-of-the-art approaches.