CLIP-based Synergistic Knowledge Transfer for Text-based Person Retrieval
This work addresses the problem of efficient and effective person retrieval from text queries for applications like surveillance, but it is incremental as it builds on existing CLIP models.
The paper tackles the challenge of bridging vision-language gaps in text-based person retrieval by introducing a CLIP-based synergistic knowledge transfer approach, which achieves state-of-the-art performance on three benchmark datasets while using only 7.4% of the model's training parameters.
Text-based Person Retrieval (TPR) aims to retrieve the target person images given a textual query. The primary challenge lies in bridging the substantial gap between vision and language modalities, especially when dealing with limited large-scale datasets. In this paper, we introduce a CLIP-based Synergistic Knowledge Transfer (CSKT) approach for TPR. Specifically, to explore the CLIP's knowledge on input side, we first propose a Bidirectional Prompts Transferring (BPT) module constructed by text-to-image and image-to-text bidirectional prompts and coupling projections. Secondly, Dual Adapters Transferring (DAT) is designed to transfer knowledge on output side of Multi-Head Attention (MHA) in vision and language. This synergistic two-way collaborative mechanism promotes the early-stage feature fusion and efficiently exploits the existing knowledge of CLIP. CSKT outperforms the state-of-the-art approaches across three benchmark datasets when the training parameters merely account for 7.4% of the entire model, demonstrating its remarkable efficiency, effectiveness and generalization.