EventFormer: A Node-graph Hierarchical Attention Transformer for Action-centric Video Event Prediction
This work addresses a gap in vision research by extending script event prediction from NLP to videos, offering a new dataset and model for a domain-specific task with potential applications in video understanding.
The authors tackled the problem of predicting subsequent events in videos, a task they term Action-centric Video Event Prediction (AVEP), by introducing a large structured dataset of 35K annotated videos and proposing EventFormer, a node-graph hierarchical attention model that outperforms existing state-of-the-art video prediction models.
Script event induction, which aims to predict the subsequent event based on the context, is a challenging task in NLP, achieving remarkable success in practical applications. However, human events are mostly recorded and presented in the form of videos rather than scripts, yet there is a lack of related research in the realm of vision. To address this problem, we introduce AVEP (Action-centric Video Event Prediction), a task that distinguishes itself from existing video prediction tasks through its incorporation of more complex logic and richer semantic information. We present a large structured dataset, which consists of about $35K$ annotated videos and more than $178K$ video clips of event, built upon existing video event datasets to support this task. The dataset offers more fine-grained annotations, where the atomic unit is represented as a multimodal event argument node, providing better structured representations of video events. Due to the complexity of event structures, traditional visual models that take patches or frames as input are not well-suited for AVEP. We propose EventFormer, a node-graph hierarchical attention based video event prediction model, which can capture both the relationships between events and their arguments and the coreferencial relationships between arguments. We conducted experiments using several SOTA video prediction models as well as LVLMs on AVEP, demonstrating both the complexity of the task and the value of the dataset. Our approach outperforms all these video prediction models. We will release the dataset and code for replicating the experiments and annotations.