Egocentric Activity Recognition and Localization on a 3D Map
This addresses the challenge of understanding human activities in 3D environments from first-person perspectives, which is incremental but important for applications in robotics and augmented reality.
The paper tackles the problem of jointly recognizing and localizing actions from egocentric videos on a known 3D map, proposing a novel deep probabilistic model that achieves strong results on both action recognition and 3D action localization across seen and unseen environments.
Given a video captured from a first person perspective and the environment context of where the video is recorded, can we recognize what the person is doing and identify where the action occurs in the 3D space? We address this challenging problem of jointly recognizing and localizing actions of a mobile user on a known 3D map from egocentric videos. To this end, we propose a novel deep probabilistic model. Our model takes the inputs of a Hierarchical Volumetric Representation (HVR) of the 3D environment and an egocentric video, infers the 3D action location as a latent variable, and recognizes the action based on the video and contextual cues surrounding its potential locations. To evaluate our model, we conduct extensive experiments on the subset of Ego4D dataset, in which both human naturalistic actions and photo-realistic 3D environment reconstructions are captured. Our method demonstrates strong results on both action recognition and 3D action localization across seen and unseen environments. We believe our work points to an exciting research direction in the intersection of egocentric vision, and 3D scene understanding.