Semantic Role Aware Correlation Transformer for Text to Video Retrieval
This addresses the problem of retrieving relevant videos from language queries for social media users, representing an incremental improvement over existing methods.
The paper tackles text-to-video retrieval by proposing a transformer that disentangles text and video into semantic roles (objects, spatial contexts, temporal contexts) to learn intra- and inter-role correlations, achieving results that surpass a current state-of-the-art method by a high margin in all metrics on YouCook2.
With the emergence of social media, voluminous video clips are uploaded every day, and retrieving the most relevant visual content with a language query becomes critical. Most approaches aim to learn a joint embedding space for plain textual and visual contents without adequately exploiting their intra-modality structures and inter-modality correlations. This paper proposes a novel transformer that explicitly disentangles the text and video into semantic roles of objects, spatial contexts and temporal contexts with an attention scheme to learn the intra- and inter-role correlations among the three roles to discover discriminative features for matching at different levels. The preliminary results on popular YouCook2 indicate that our approach surpasses a current state-of-the-art method, with a high margin in all metrics. It also overpasses two SOTA methods in terms of two metrics.