Event-Aware Instructed Assistant for Referring Video Segmentation
For researchers in video understanding, this work introduces a novel event-aware decomposition approach to improve referring video segmentation, though it is incremental as it builds on existing MLLM frameworks.
The paper addresses the problem that existing referring video segmentation methods treat a video as a single event, leading to confusion and hallucinations. The proposed EVIS method decomposes videos into simple events using learnable Event Queries and achieves strong performance on 5 public benchmarks.
Existing referring video segmentation methods often treat a video as a single event consisting of multiple images, overlooking the fact that a video typically contains multiple distinct events. Under such a mechanism, the model needs to directly understand all the complex content in the video and text, which can easily lead to confusion and hallucinations. To address this issue, we propose to decompose a video to a set of simple events by learnable Event Query, and understand complex video content in an event-by-event, easy-to-understand manner. This is based on the observation that natural language expressions often divide a video into distinct, text-related segments, each representing a separate event within a compound event. We introduce EVIS, an Event-Aware Video Instructed Segmentation Assistant, which utilizes text-guided Event Queries to partition a video into simple events, extracting event-aware visual-text features to achieve a hierarchical understanding of the video. Additionally, we propose Object-Pixel-Hybrid Learning, which enables the MLLMs to track targets in long-term videos by integrating fine-grained pixel features with prior object queries. Extensive experimental results on 5 public benchmarks demonstrate EVIS's strong performance in addressing the referring video segmentation task.