Vision-aware Multimodal Prompt Tuning for Uploadable Multi-source Few-shot Domain Adaptation
This work addresses low-resource edge computing scenarios for decentralized collaborative learning, though it appears incremental as it builds on existing prompt tuning approaches.
The paper tackles the challenge of reducing computational load on edge devices in multi-source few-shot domain adaptation by proposing a vision-aware multimodal prompt tuning framework (VAMP) under an uploadable schema, achieving superior performance over previous methods on OfficeHome and DomainNet datasets.
Conventional multi-source domain few-shot adaptation (MFDA) faces the challenge of further reducing the load on edge-side devices in low-resource scenarios. Considering the native language-supervised advantage of CLIP and the plug-and-play nature of prompt to transfer CLIP efficiently, this paper introduces an uploadable multi-source few-shot domain adaptation (UMFDA) schema. It belongs to a decentralized edge collaborative learning in the edge-side models that must maintain a low computational load. And only a limited amount of annotations in source domain data is provided, with most of the data being unannotated. Further, this paper proposes a vision-aware multimodal prompt tuning framework (VAMP) under the decentralized schema, where the vision-aware prompt guides the text domain-specific prompt to maintain semantic discriminability and perceive the domain information. The cross-modal semantic and domain distribution alignment losses optimize each edge-side model, while text classifier consistency and semantic diversity losses promote collaborative learning among edge-side models. Extensive experiments were conducted on OfficeHome and DomainNet datasets to demonstrate the effectiveness of the proposed VAMP in the UMFDA, which outperformed the previous prompt tuning methods.