GPTSee: Enhancing Moment Retrieval and Highlight Detection via Description-Based Similarity Features
This work addresses video analysis tasks for applications like content search and summarization, presenting an incremental improvement by combining LLMs with existing transformer architectures.
The paper tackled moment retrieval and highlight detection in videos by integrating large language models (LLMs) to generate descriptions and compute similarity features, achieving state-of-the-art results with improved positioning accuracy over methods like Moment-DETR.
Moment retrieval (MR) and highlight detection (HD) aim to identify relevant moments and highlights in video from corresponding natural language query. Large language models (LLMs) have demonstrated proficiency in various computer vision tasks. However, existing methods for MR\&HD have not yet been integrated with LLMs. In this letter, we propose a novel two-stage model that takes the output of LLMs as the input to the second-stage transformer encoder-decoder. First, MiniGPT-4 is employed to generate the detailed description of the video frame and rewrite the query statement, fed into the encoder as new features. Then, semantic similarity is computed between the generated description and the rewritten queries. Finally, continuous high-similarity video frames are converted into span anchors, serving as prior position information for the decoder. Experiments demonstrate that our approach achieves a state-of-the-art result, and by using only span anchors and similarity scores as outputs, positioning accuracy outperforms traditional methods, like Moment-DETR.