Video Moment Localization using Object Evidence and Reverse Captioning
This work addresses the challenge of precisely locating video moments based on natural language queries, which is important for video retrieval and analysis applications, but it is incremental as it extends an existing model.
The paper tackles the problem of language-based temporal localization of moments in untrimmed videos, where queries lack predefined classes and may be complex, and reports that their proposed MML model outperforms the MAC baseline by 4.93% and 1.70% on R@1 and R@5 metrics on the Charades-STA dataset.
We address the problem of language-based temporal localization of moments in untrimmed videos. Compared to temporal localization with fixed categories, this problem is more challenging as the language-based queries have no predefined activity classes and may also contain complex descriptions. Current state-of-the-art model MAC addresses it by mining activity concepts from both video and language modalities. This method encodes the semantic activity concepts from the verb/object pair in a language query and leverages visual activity concepts from video activity classification prediction scores. We propose "Multi-faceted VideoMoment Localizer" (MML), an extension of MAC model by the introduction of visual object evidence via object segmentation masks and video understanding features via video captioning. Furthermore, we improve language modelling in sentence embedding. We experimented on Charades-STA dataset and identified that MML outperforms MAC baseline by 4.93% and 1.70% on R@1 and R@5metrics respectively. Our code and pre-trained model are publicly available at https://github.com/madhawav/MML.