Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications
This work addresses the need for detailed hand-object interaction modeling in egocentric vision, enabling applications in behavior analysis and AR/VR, though it is incremental as it builds on existing segmentation techniques.
The authors tackled the problem of fine-grained hand-object segmentation in egocentric videos by introducing a new dataset with 11,243 labeled images and a context-aware augmentation method, resulting in a model that improves downstream applications like activity recognition and 3D reconstruction.
Egocentric videos offer fine-grained information for high-fidelity modeling of human behaviors. Hands and interacting objects are one crucial aspect of understanding a viewer's behaviors and intentions. We provide a labeled dataset consisting of 11,243 egocentric images with per-pixel segmentation labels of hands and objects being interacted with during a diverse array of daily activities. Our dataset is the first to label detailed hand-object contact boundaries. We introduce a context-aware compositional data augmentation technique to adapt to out-of-distribution YouTube egocentric video. We show that our robust hand-object segmentation model and dataset can serve as a foundational tool to boost or enable several downstream vision applications, including hand state classification, video activity recognition, 3D mesh reconstruction of hand-object interactions, and video inpainting of hand-object foregrounds in egocentric videos. Dataset and code are available at: https://github.com/owenzlz/EgoHOS