Towards Effective Long-Video Event Prediction via Multi-Level Event Semantics Mining
This work is significant for researchers and developers working on long-video content understanding and automated decision-making, as it provides an effective method for predicting future events in complex, extended video narratives, an area where current models fall short.
The paper addresses the challenge of predicting future events in long videos, a task where existing Long-Video Language Models struggle due to their inability to extract precise event details and analyze event development. The proposed VISTA framework tackles this by employing a character-centric visual prompt for detail extraction, a knowledge-enhanced iterative retrieval for constructing event chains, and a human-like propose-then-retrieve strategy for robust predictions. Experiments on real-world datasets confirm VISTA's effectiveness.
Accurately predicting future events is fundamental to content understanding and decision-making across various domains. While prior research has primarily focused on text or short-video scenarios, long-video event prediction, characterized by vast multimodal context and more complex narratives, remains underexplored. Meanwhile, although recent Long-Video Language Models (LVLMs), built on Large Language Models (LLMs) and Vision-Language Models (VLMs), have shown promise in long-video question answering and summarization, they struggle to generalize to event prediction, as they can neither precisely extract event-related details nor perform fine-grained analysis of event development. To address this gap, we propose VISTA, a multi-level event semantics mining framework for long-video event prediction. Initially, VISTA applies a character-centric visual prompt to precisely extract event-related visual details, enhancing detail-level semantics; subsequently, it employs a knowledge-enhanced iterative retrieval strategy, guiding the LLM to progressively construct logically coherent event chains, thereby improving event-level narratives; ultimately, VISTA adopts a human-like propose-then-retrieve strategy to generate diverse future-oriented proposals and integrate multi-level clues, producing robust and accurate predictions. Extensive experiments on real-world datasets validate the effectiveness of VISTA for long-video event prediction.