H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions
This addresses the challenge of egocentric recognition for applications like robotics and AR/VR, though it appears incremental as it builds on existing pose estimation and interaction modeling techniques.
The paper tackles the problem of understanding 3D hand and object interactions from egocentric RGB images by proposing a unified framework that jointly estimates poses, models interactions, and recognizes object and action classes in a single feed-forward pass, achieving state-of-the-art performance compared to methods using depth data and ground-truth annotations.
We present a unified framework for understanding 3D hand and object interactions in raw image sequences from egocentric RGB cameras. Given a single RGB image, our model jointly estimates the 3D hand and object poses, models their interactions, and recognizes the object and action classes with a single feed-forward pass through a neural network. We propose a single architecture that does not rely on external detection algorithms but rather is trained end-to-end on single images. We further merge and propagate information in the temporal domain to infer interactions between hand and object trajectories and recognize actions. The complete model takes as input a sequence of frames and outputs per-frame 3D hand and object pose predictions along with the estimates of object and action categories for the entire sequence. We demonstrate state-of-the-art performance of our algorithm even in comparison to the approaches that work on depth data and ground-truth annotations.