BiLMa: Bidirectional Local-Matching for Text-based Person Re-identification
This work addresses a crucial challenge in text-based person re-identification for improving retrieval accuracy, representing an incremental advancement by adding bidirectional matching to existing methods.
The paper tackles the problem of aligning images and texts globally and locally in text-based person re-identification by introducing a Bidirectional Local-Matching framework that jointly optimizes Masked Language Modeling and Masked Image Modeling, achieving state-of-the-art Rank@1 and mAP scores on three benchmarks.
Text-based person re-identification (TBPReID) aims to retrieve person images represented by a given textual query. In this task, how to effectively align images and texts globally and locally is a crucial challenge. Recent works have obtained high performances by solving Masked Language Modeling (MLM) to align image/text parts. However, they only performed uni-directional (i.e., from image to text) local-matching, leaving room for improvement by introducing opposite-directional (i.e., from text to image) local-matching. In this work, we introduce Bidirectional Local-Matching (BiLMa) framework that jointly optimize MLM and Masked Image Modeling (MIM) in TBPReID model training. With this framework, our model is trained so as the labels of randomly masked both image and text tokens are predicted by unmasked tokens. In addition, to narrow the semantic gap between image and text in MIM, we propose Semantic MIM (SemMIM), in which the labels of masked image tokens are automatically given by a state-of-the-art human parser. Experimental results demonstrate that our BiLMa framework with SemMIM achieves state-of-the-art Rank@1 and mAP scores on three benchmarks.