Egocentric Video Task Translation
This addresses the need for integrated video understanding in wearable cameras, offering a novel method to handle interconnected tasks like hand-object manipulations and navigation, though it is incremental in building on existing models.
The paper tackles the problem of isolated video understanding tasks in egocentric video by proposing EgoTask Translation (EgoT2), a unified approach that translates outputs from separate task-specific models to improve performance across multiple tasks simultaneously, achieving top-ranked results on four Ego4D 2022 benchmark challenges.
Different video understanding tasks are typically treated in isolation, and even with distinct types of curated data (e.g., classifying sports in one dataset, tracking animals in another). However, in wearable cameras, the immersive egocentric perspective of a person engaging with the world around them presents an interconnected web of video understanding tasks -- hand-object manipulations, navigation in the space, or human-human interactions -- that unfold continuously, driven by the person's goals. We argue that this calls for a much more unified approach. We propose EgoTask Translation (EgoT2), which takes a collection of models optimized on separate tasks and learns to translate their outputs for improved performance on any or all of them at once. Unlike traditional transfer or multi-task learning, EgoT2's flipped design entails separate task-specific backbones and a task translator shared across all tasks, which captures synergies between even heterogeneous tasks and mitigates task competition. Demonstrating our model on a wide array of video tasks from Ego4D, we show its advantages over existing transfer paradigms and achieve top-ranked results on four of the Ego4D 2022 benchmark challenges.