SOS! Self-supervised Learning Over Sets Of Handled Objects In Egocentric Action Recognition
This addresses the problem of distractors in egocentric video recognition for applications like robotics or AR, offering a flexible integration method without retraining action models, though it is incremental as it builds on existing object detection and self-supervised learning.
The paper tackles the challenge of integrating object information into egocentric action recognition by introducing SOS, a self-supervised method to pre-train an Objects In Contact representation model, which boosts performance on EPIC-KITCHENS-100 and EGTEA datasets.
Learning an egocentric action recognition model from video data is challenging due to distractors (e.g., irrelevant objects) in the background. Further integrating object information into an action model is hence beneficial. Existing methods often leverage a generic object detector to identify and represent the objects in the scene. However, several important issues remain. Object class annotations of good quality for the target domain (dataset) are still required for learning good object representation. Besides, previous methods deeply couple the existing action models and need to retrain them jointly with object representation, leading to costly and inflexible integration. To overcome both limitations, we introduce Self-Supervised Learning Over Sets (SOS), an approach to pre-train a generic Objects In Contact (OIC) representation model from video object regions detected by an off-the-shelf hand-object contact detector. Instead of augmenting object regions individually as in conventional self-supervised learning, we view the action process as a means of natural data transformations with unique spatio-temporal continuity and exploit the inherent relationships among per-video object sets. Extensive experiments on two datasets, EPIC-KITCHENS-100 and EGTEA, show that our OIC significantly boosts the performance of multiple state-of-the-art video classification models.