Jointly Visual- and Semantic-Aware Graph Memory Networks for Temporal Sentence Localization in Videos
This addresses limitations in video localization for applications like video retrieval, though it appears incremental by combining existing visual and semantic approaches.
The paper tackles temporal sentence localization in videos by proposing a Hierarchical Visual- and Semantic-Aware Reasoning Network (HVSARN) that integrates both visual and semantic reasoning, achieving state-of-the-art performance on three datasets.
Temporal sentence localization in videos (TSLV) aims to retrieve the most interested segment in an untrimmed video according to a given sentence query. However, almost of existing TSLV approaches suffer from the same limitations: (1) They only focus on either frame-level or object-level visual representation learning and corresponding correlation reasoning, but fail to integrate them both; (2) They neglect to leverage the rich semantic contexts to further benefit the query reasoning. To address these issues, in this paper, we propose a novel Hierarchical Visual- and Semantic-Aware Reasoning Network (HVSARN), which enables both visual- and semantic-aware query reasoning from object-level to frame-level. Specifically, we present a new graph memory mechanism to perform visual-semantic query reasoning: For visual reasoning, we design a visual graph memory to leverage visual information of video; For semantic reasoning, a semantic graph memory is also introduced to explicitly leverage semantic knowledge contained in the classes and attributes of video objects, and perform correlation reasoning in the semantic space. Experiments on three datasets demonstrate that our HVSARN achieves a new state-of-the-art performance.