Video-adverb retrieval with compositional adverb-action embeddings
This addresses fine-grained video understanding for applications like video analysis and retrieval, though it is incremental as it builds on existing retrieval frameworks.
The paper tackles video-to-adverb retrieval by aligning video embeddings with compositional adverb-action text embeddings using a residual gating mechanism and a novel training objective, achieving state-of-the-art performance on five benchmarks and outperforming prior works for unseen adverb-action compositions.
Retrieving adverbs that describe an action in a video poses a crucial step towards fine-grained video understanding. We propose a framework for video-to-adverb retrieval (and vice versa) that aligns video embeddings with their matching compositional adverb-action text embedding in a joint embedding space. The compositional adverb-action text embedding is learned using a residual gating mechanism, along with a novel training objective consisting of triplet losses and a regression target. Our method achieves state-of-the-art performance on five recent benchmarks for video-adverb retrieval. Furthermore, we introduce dataset splits to benchmark video-adverb retrieval for unseen adverb-action compositions on subsets of the MSR-VTT Adverbs and ActivityNet Adverbs datasets. Our proposed framework outperforms all prior works for the generalisation task of retrieving adverbs from videos for unseen adverb-action compositions. Code and dataset splits are available at https://hummelth.github.io/ReGaDa/.