DM-Adapter: Domain-Aware Mixture-of-Adapters for Text-Based Person Retrieval
This work addresses efficiency and feature extraction issues in text-based person retrieval, an incremental advancement for practical applications like surveillance and security.
The paper tackles the challenges of computationally expensive fine-tuning and lack of fine-grained feature extraction in text-based person retrieval by proposing DM-Adapter, which unifies Mixture-of-Experts and parameter-efficient transfer learning to enhance feature representations; it achieves state-of-the-art performance with significant improvements over previous methods.
Text-based person retrieval (TPR) has gained significant attention as a fine-grained and challenging task that closely aligns with practical applications. Tailoring CLIP to person domain is now a emerging research topic due to the abundant knowledge of vision-language pretraining, but challenges still remain during fine-tuning: (i) Previous full-model fine-tuning in TPR is computationally expensive and prone to overfitting.(ii) Existing parameter-efficient transfer learning (PETL) for TPR lacks of fine-grained feature extraction. To address these issues, we propose Domain-Aware Mixture-of-Adapters (DM-Adapter), which unifies Mixture-of-Experts (MOE) and PETL to enhance fine-grained feature representations while maintaining efficiency. Specifically, Sparse Mixture-of-Adapters is designed in parallel to MLP layers in both vision and language branches, where different experts specialize in distinct aspects of person knowledge to handle features more finely. To promote the router to exploit domain information effectively and alleviate the routing imbalance, Domain-Aware Router is then developed by building a novel gating function and injecting learnable domain-aware prompts. Extensive experiments show that our DM-Adapter achieves state-of-the-art performance, outperforming previous methods by a significant margin.