ProTA: Probabilistic Token Aggregation for Text-Video Retrieval
It improves retrieval accuracy for text-video systems, but is incremental as it builds on existing methods for handling asymmetric content.
The paper tackles the problem of text-video retrieval by addressing content asymmetry between video clips and captions, proposing Probabilistic Token Aggregation (ProTA) to handle cross-modal interactions, and achieves significant improvements with results like 50.9% on MSR-VTT, 25.8% on LSMDC, and 47.2% on DiDeMo.
Text-video retrieval aims to find the most relevant cross-modal samples for a given query. Recent methods focus on modeling the whole spatial-temporal relations. However, since video clips contain more diverse content than captions, the model aligning these asymmetric video-text pairs has a high risk of retrieving many false positive results. In this paper, we propose Probabilistic Token Aggregation (ProTA) to handle cross-modal interaction with content asymmetry. Specifically, we propose dual partial-related aggregation to disentangle and re-aggregate token representations in both low-dimension and high-dimension spaces. We propose token-based probabilistic alignment to generate token-level probabilistic representation and maintain the feature representation diversity. In addition, an adaptive contrastive loss is proposed to learn compact cross-modal distribution space. Based on extensive experiments, ProTA achieves significant improvements on MSR-VTT (50.9%), LSMDC (25.8%), and DiDeMo (47.2%).