Fine-grained Spatiotemporal Grounding on Egocentric Videos
This work addresses the underexplored egocentric video grounding problem, which is important for applications like augmented reality and robotics, by providing essential resources and insights, though it is incremental as it builds on existing methods with new data.
The paper tackles the problem of spatiotemporal video grounding in egocentric videos by introducing EgoMask, a pixel-level benchmark, and EgoMask-Train, a large-scale training dataset, revealing that state-of-the-art models perform poorly on this benchmark but fine-tuning on EgoMask-Train yields significant improvements while maintaining performance on exocentric datasets.
Spatiotemporal video grounding aims to localize target entities in videos based on textual queries. While existing research has made significant progress in exocentric videos, the egocentric setting remains relatively underexplored, despite its growing importance in applications such as augmented reality and robotics. In this work, we conduct a systematic analysis of the discrepancies between egocentric and exocentric videos, revealing key challenges such as shorter object durations, sparser trajectories, smaller object sizes, and larger positional shifts. To address these challenges, we introduce EgoMask, the first pixel-level benchmark for fine-grained spatiotemporal grounding in egocentric videos. It is constructed by our proposed automatic annotation pipeline, which annotates referring expressions and object masks across short-, medium-, and long-term videos. Additionally, we create EgoMask-Train, a large-scale training dataset to facilitate model development. Experiments demonstrate that the state-of-the-art spatiotemporal grounding models perform poorly on our benchmark EgoMask, but fine-tuning on EgoMask-Train yields significant improvements, while preserving performance on exocentric datasets. Our work thus provides essential resources and insights for advancing egocentric video understanding. Our code is available at https://github.com/LaVi-Lab/EgoMask .