Open-Vocabulary Video Relation Extraction
This addresses the need for deeper video understanding beyond general action classification, focusing on actors and relationships, though it is incremental as it builds on existing cross-modal generation models.
The paper tackles the problem of superficial video understanding by introducing Open-vocabulary Video Relation Extraction (OVRE), a task that extracts action-centric relation triplets described in natural language, and curates the Moments-OVRE dataset with 180K videos to benchmark models.
A comprehensive understanding of videos is inseparable from describing the action with its contextual action-object interactions. However, many current video understanding tasks prioritize general action classification and overlook the actors and relationships that shape the nature of the action, resulting in a superficial understanding of the action. Motivated by this, we introduce Open-vocabulary Video Relation Extraction (OVRE), a novel task that views action understanding through the lens of action-centric relation triplets. OVRE focuses on pairwise relations that take part in the action and describes these relation triplets with natural languages. Moreover, we curate the Moments-OVRE dataset, which comprises 180K videos with action-centric relation triplets, sourced from a multi-label action classification dataset. With Moments-OVRE, we further propose a crossmodal mapping model to generate relation triplets as a sequence. Finally, we benchmark existing cross-modal generation models on the new task of OVRE.