Enhancing Next Active Object-based Egocentric Action Anticipation with Guided Attention
This addresses the problem of anticipating actions in egocentric videos for applications like robotics and AR, but it is incremental as it builds on existing methods by focusing on object interactions.
The paper tackles short-term action anticipation in first-person videos by introducing a guided attention mechanism between objects and spatiotemporal features, resulting in GANO outperforming state-of-the-art methods on the largest egocentric dataset for predicting next active object labels, bounding boxes, future actions, and time to contact.
Short-term action anticipation (STA) in first-person videos is a challenging task that involves understanding the next active object interactions and predicting future actions. Existing action anticipation methods have primarily focused on utilizing features extracted from video clips, but often overlooked the importance of objects and their interactions. To this end, we propose a novel approach that applies a guided attention mechanism between the objects, and the spatiotemporal features extracted from video clips, enhancing the motion and contextual information, and further decoding the object-centric and motion-centric information to address the problem of STA in egocentric videos. Our method, GANO (Guided Attention for Next active Objects) is a multi-modal, end-to-end, single transformer-based network. The experimental results performed on the largest egocentric dataset demonstrate that GANO outperforms the existing state-of-the-art methods for the prediction of the next active object label, its bounding box location, the corresponding future action, and the time to contact the object. The ablation study shows the positive contribution of the guided attention mechanism compared to other fusion methods. Moreover, it is possible to improve the next active object location and class label prediction results of GANO by just appending the learnable object tokens with the region of interest embeddings.