GEA: Generation-Enhanced Alignment for Text-to-Image Person Retrieval
This work addresses cross-modal retrieval challenges in computer vision for applications like surveillance or security, but it is incremental as it builds on existing methods with a generative enhancement.
The paper tackles the problem of text-to-image person retrieval, where textual queries often fail to accurately reflect image content, leading to poor cross-modal alignment and overfitting; the proposed Generation-Enhanced Alignment method improves retrieval performance, achieving competitive results on public datasets like CUHK-PEDES, RSTPReid, and ICFG-PEDES.
Text-to-Image Person Retrieval (TIPR) aims to retrieve person images based on natural language descriptions. Although many TIPR methods have achieved promising results, sometimes textual queries cannot accurately and comprehensively reflect the content of the image, leading to poor cross-modal alignment and overfitting to limited datasets. Moreover, the inherent modality gap between text and image further amplifies these issues, making accurate cross-modal retrieval even more challenging. To address these limitations, we propose the Generation-Enhanced Alignment (GEA) from a generative perspective. GEA contains two parallel modules: (1) Text-Guided Token Enhancement (TGTE), which introduces diffusion-generated images as intermediate semantic representations to bridge the gap between text and visual patterns. These generated images enrich the semantic representation of text and facilitate cross-modal alignment. (2) Generative Intermediate Fusion (GIF), which combines cross-attention between generated images, original images, and text features to generate a unified representation optimized by triplet alignment loss. We conduct extensive experiments on three public TIPR datasets, CUHK-PEDES, RSTPReid, and ICFG-PEDES, to evaluate the performance of GEA. The results justify the effectiveness of our method. More implementation details and extended results are available at https://github.com/sugelamyd123/Sup-for-GEA.