A Backpack Full of Skills: Egocentric Video Understanding with Diverse Task Perspectives
This work addresses the challenge of enabling robots to efficiently understand and learn from egocentric videos, though it appears incremental as it builds on existing benchmarks and methods.
The paper tackled the problem of holistic video understanding by proposing EgoPack, a unified approach that learns and transfers task perspectives across multiple downstream tasks, achieving state-of-the-art results on four Ego4D benchmarks.
Human comprehension of a video stream is naturally broad: in a few instants, we are able to understand what is happening, the relevance and relationship of objects, and forecast what will follow in the near future, everything all at once. We believe that - to effectively transfer such an holistic perception to intelligent machines - an important role is played by learning to correlate concepts and to abstract knowledge coming from different tasks, to synergistically exploit them when learning novel skills. To accomplish this, we seek for a unified approach to video understanding which combines shared temporal modelling of human actions with minimal overhead, to support multiple downstream tasks and enable cooperation when learning novel skills. We then propose EgoPack, a solution that creates a collection of task perspectives that can be carried across downstream tasks and used as a potential source of additional insights, as a backpack of skills that a robot can carry around and use when needed. We demonstrate the effectiveness and efficiency of our approach on four Ego4D benchmarks, outperforming current state-of-the-art methods.