Egocentric Prediction of Action Target in 3D
This addresses a gap in human-robot collaboration by providing a new dataset and benchmarks for an understudied egocentric vision task.
The paper tackles the problem of predicting the target location of a person's object manipulation action in 3D from egocentric vision as early as possible, and introduces a large multimodality dataset of over 1 million frames with baseline methods showing the task's feasibility.
We are interested in anticipating as early as possible the target location of a person's object manipulation action in a 3D workspace from egocentric vision. It is important in fields like human-robot collaboration, but has not yet received enough attention from vision and learning communities. To stimulate more research on this challenging egocentric vision task, we propose a large multimodality dataset of more than 1 million frames of RGB-D and IMU streams, and provide evaluation metrics based on our high-quality 2D and 3D labels from semi-automatic annotation. Meanwhile, we design baseline methods using recurrent neural networks and conduct various ablation studies to validate their effectiveness. Our results demonstrate that this new task is worthy of further study by researchers in robotics, vision, and learning communities.