Joint Moment Retrieval and Highlight Detection Via Natural Language Queries
This work addresses video summarization for users needing efficient content navigation, but it appears incremental as it builds on existing transformer techniques.
The paper tackles the problem of retrieving relevant video moments and detecting highlights based on natural language queries, proposing a multi-modal transformer method that uses visual and audio cues, and demonstrates its flexibility on datasets like YouTube Highlights and TVSum.
Video summarization has become an increasingly important task in the field of computer vision due to the vast amount of video content available on the internet. In this project, we propose a new method for natural language query based joint video summarization and highlight detection using multi-modal transformers. This approach will use both visual and audio cues to match a user's natural language query to retrieve the most relevant and interesting moments from a video. Our approach employs multiple recent techniques used in Vision Transformers (ViTs) to create a transformer-like encoder-decoder model. We evaluated our approach on multiple datasets such as YouTube Highlights and TVSum to demonstrate the flexibility of our proposed method.