Denoise-then-Retrieve: Text-Conditioned Video Denoising for Video Moment Retrieval
This addresses a specific bottleneck in video moment retrieval for applications like video search and analysis, though it is incremental as it builds on existing VMR models.
The paper tackles the problem of irrelevant video clips disrupting multimodal alignment in text-driven Video Moment Retrieval by proposing a denoise-then-retrieve paradigm that filters text-irrelevant clips, resulting in state-of-the-art performance on Charades-STA and QVHighlights datasets.
Current text-driven Video Moment Retrieval (VMR) methods encode all video clips, including irrelevant ones, disrupting multimodal alignment and hindering optimization. To this end, we propose a denoise-then-retrieve paradigm that explicitly filters text-irrelevant clips from videos and then retrieves the target moment using purified multimodal representations. Following this paradigm, we introduce the Denoise-then-Retrieve Network (DRNet), comprising Text-Conditioned Denoising (TCD) and Text-Reconstruction Feedback (TRF) modules. TCD integrates cross-attention and structured state space blocks to dynamically identify noisy clips and produce a noise mask to purify multimodal video representations. TRF further distills a single query embedding from purified video representations and aligns it with the text embedding, serving as auxiliary supervision for denoising during training. Finally, we perform conditional retrieval using text embeddings on purified video representations for accurate VMR. Experiments on Charades-STA and QVHighlights demonstrate that our approach surpasses state-of-the-art methods on all metrics. Furthermore, our denoise-then-retrieve paradigm is adaptable and can be seamlessly integrated into advanced VMR models to boost performance.