Is an Object-Centric Video Representation Beneficial for Transfer?
This work addresses the challenge of enhancing video model transferability for computer vision applications, though it appears incremental as it builds on existing transformer architectures with specific modifications.
The paper tackled the problem of learning object-centric video representations to improve transferability to novel tasks beyond action classification, and demonstrated that their model outperforms prior video representations on tasks like classifying actions on unseen objects and environments, low-shot learning, and linear probing to downstream tasks.
The objective of this work is to learn an object-centric video representation, with the aim of improving transferability to novel tasks, i.e., tasks different from the pre-training task of action classification. To this end, we introduce a new object-centric video recognition model based on a transformer architecture. The model learns a set of object-centric summary vectors for the video, and uses these vectors to fuse the visual and spatio-temporal trajectory 'modalities' of the video clip. We also introduce a novel trajectory contrast loss to further enhance objectness in these summary vectors. With experiments on four datasets -- SomethingSomething-V2, SomethingElse, Action Genome and EpicKitchens -- we show that the object-centric model outperforms prior video representations (both object-agnostic and object-aware), when: (1) classifying actions on unseen objects and unseen environments; (2) low-shot learning of novel classes; (3) linear probe to other downstream tasks; as well as (4) for standard action classification.