HiERO: understanding the hierarchy of human behavior enhances reasoning on egocentric videos
This addresses the problem of improving video understanding for AI systems in egocentric vision, with incremental advancements in leveraging hierarchical patterns.
The paper tackled the challenge of reasoning about complex human activities in egocentric videos by leveraging their underlying hierarchical structure, achieving state-of-the-art performance on multiple benchmarks and outperforming fully-supervised methods by 12.5% F1 on EgoProceL in zero-shot.
Human activities are particularly complex and variable, and this makes challenging for deep learning models to reason about them. However, we note that such variability does have an underlying structure, composed of a hierarchy of patterns of related actions. We argue that such structure can emerge naturally from unscripted videos of human activities, and can be leveraged to better reason about their content. We present HiERO, a weakly-supervised method to enrich video segments features with the corresponding hierarchical activity threads. By aligning video clips with their narrated descriptions, HiERO infers contextual, semantic and temporal reasoning with an hierarchical architecture. We prove the potential of our enriched features with multiple video-text alignment benchmarks (EgoMCQ, EgoNLQ) with minimal additional training, and in zero-shot for procedure learning tasks (EgoProceL and Ego4D Goal-Step). Notably, HiERO achieves state-of-the-art performance in all the benchmarks, and for procedure learning tasks it outperforms fully-supervised methods by a large margin (+12.5% F1 on EgoProceL) in zero shot. Our results prove the relevance of using knowledge of the hierarchy of human activities for multiple reasoning tasks in egocentric vision.