Robust Egocentric Visual Attention Prediction Through Language-guided Scene Context-aware Learning
This addresses the problem of analyzing where people look in first-person videos, which is incremental as it builds on existing methods by incorporating language guidance for better context awareness.
The paper tackles the challenge of predicting visual attention in egocentric videos by proposing a language-guided scene context-aware learning framework, achieving state-of-the-art performance on Ego4D and AEA datasets with enhanced robustness.
As the demand for analyzing egocentric videos grows, egocentric visual attention prediction, anticipating where a camera wearer will attend, has garnered increasing attention. However, it remains challenging due to the inherent complexity and ambiguity of dynamic egocentric scenes. Motivated by evidence that scene contextual information plays a crucial role in modulating human attention, in this paper, we present a language-guided scene context-aware learning framework for robust egocentric visual attention prediction. We first design a context perceiver which is guided to summarize the egocentric video based on a language-based scene description, generating context-aware video representations. We then introduce two training objectives that: 1) encourage the framework to focus on the target point-of-interest regions and 2) suppress distractions from irrelevant regions which are less likely to attract first-person attention. Extensive experiments on Ego4D and Aria Everyday Activities (AEA) datasets demonstrate the effectiveness of our approach, achieving state-of-the-art performance and enhanced robustness across diverse, dynamic egocentric scenarios.