Whareformer: Learning to Track What is Where in Long Egocentric Videos
This work addresses the problem of online object tracking in egocentric videos for computer vision researchers, providing a novel learning-based approach that outperforms prior methods.
Whareformer is the first learning-based solution for tracking objects in long egocentric videos, maintaining instance locations even when objects leave the field of view. It achieves state-of-the-art performance on three datasets, with significant absolute improvements over prior work.
The recently established 'Out of Sight, Not out of Mind' (OSNOM) task for egocentric videos focuses on tracking objects that are moved by the camera wearer, online, maintaining knowledge of instance locations throughout the video even when they leave the field of view or become heavily occluded. In this paper, we propose the first learning-based solution to the OSNOM task: Whareformer, a transformer-based model with two components: an updatable memory of established tracks and a track assignment module that associates observations with existing tracks in a feed-forward manner. Whareformer jointly reasons over evolving object appearance (what) and updated 3D location (where), and employs a dedicated New Track token to reason about novel objects. Thanks to its design choices of using relative distances and evolving track representations, Whareformer is trained on a small set of 56 videos but achieves SOTA performance on 260 long test videos from three datasets: EPIC-KITCHENS-100 (unseen videos), IT3DEgo, and HD-EPIC, with significant absolute improvements over prior work.