MmAP : Multi-modal Alignment Prompt for Cross-domain Multi-task Learning
This work addresses the problem of efficient multi-task learning for researchers and practitioners in computer vision and NLP, offering a parameter-efficient method that is incremental but novel in its multi-modal alignment approach.
The paper tackles the challenge of increasing decoder complexity in multi-task learning by integrating the CLIP vision-language model and proposing a Multi-modal Alignment Prompt (MmAP) that aligns text and visual modalities during fine-tuning, achieving significant performance improvements with only about 0.09% of trainable parameters compared to full fine-tuning.
Multi-Task Learning (MTL) is designed to train multiple correlated tasks simultaneously, thereby enhancing the performance of individual tasks. Typically, a multi-task network structure consists of a shared backbone and task-specific decoders. However, the complexity of the decoders increases with the number of tasks. To tackle this challenge, we integrate the decoder-free vision-language model CLIP, which exhibits robust zero-shot generalization capability. Recently, parameter-efficient transfer learning methods have been extensively explored with CLIP for adapting to downstream tasks, where prompt tuning showcases strong potential. Nevertheless, these methods solely fine-tune a single modality (text or visual), disrupting the modality structure of CLIP. In this paper, we first propose Multi-modal Alignment Prompt (MmAP) for CLIP, which aligns text and visual modalities during fine-tuning process. Building upon MmAP, we develop an innovative multi-task prompt learning framework. On the one hand, to maximize the complementarity of tasks with high similarity, we utilize a gradient-driven task grouping method that partitions tasks into several disjoint groups and assign a group-shared MmAP to each group. On the other hand, to preserve the unique characteristics of each task, we assign an task-specific MmAP to each task. Comprehensive experiments on two large multi-task learning datasets demonstrate that our method achieves significant performance improvements compared to full fine-tuning while only utilizing approximately 0.09% of trainable parameters.