Mutual Context Network for Jointly Estimating Egocentric Gaze and Actions
This addresses the problem of understanding human behavior in egocentric videos for applications like robotics or assistive technology, representing an incremental improvement through joint modeling.
The paper tackles the coupled tasks of gaze prediction and action recognition in egocentric videos by exploring their mutual context, proposing a mutual context network (MCN) that jointly learns these tasks end-to-end, achieving state-of-the-art performance on public datasets.
In this work, we address two coupled tasks of gaze prediction and action recognition in egocentric videos by exploring their mutual context. Our assumption is that in the procedure of performing a manipulation task, what a person is doing determines where the person is looking at, and the gaze point reveals gaze and non-gaze regions which contain important and complementary information about the undergoing action. We propose a novel mutual context network (MCN) that jointly learns action-dependent gaze prediction and gaze-guided action recognition in an end-to-end manner. Experiments on public egocentric video datasets demonstrate that our MCN achieves state-of-the-art performance of both gaze prediction and action recognition.