Cross-modal Manifold Cutmix for Self-supervised Video Representation Learning
This work addresses the data scarcity issue in self-supervised video representation learning for applications like action recognition and retrieval, though it is incremental as it builds on existing augmentation and mixing techniques.
The paper tackles the problem of expensive large-scale video data acquisition for self-supervised learning by proposing a video augmentation method called STC-mix, which combines preliminary video mixing with cross-modal feature insertion, resulting in performance on par with other approaches on action recognition and video retrieval tasks while requiring less training data.
Contrastive representation learning of videos highly relies on the availability of millions of unlabelled videos. This is practical for videos available on web but acquiring such large scale of videos for real-world applications is very expensive and laborious. Therefore, in this paper we focus on designing video augmentation for self-supervised learning, we first analyze the best strategy to mix videos to create a new augmented video sample. Then, the question remains, can we make use of the other modalities in videos for data mixing? To this end, we propose Cross-Modal Manifold Cutmix (CMMC) that inserts a video tesseract into another video tesseract in the feature space across two different modalities. We find that our video mixing strategy STC-mix, i.e. preliminary mixing of videos followed by CMMC across different modalities in a video, improves the quality of learned video representations. We conduct thorough experiments for two downstream tasks: action recognition and video retrieval on two small scale video datasets UCF101, and HMDB51. We also demonstrate the effectiveness of our STC-mix on NTU dataset where domain knowledge is limited. We show that the performance of our STC-mix on both the downstream tasks is on par with the other self-supervised approaches while requiring less training data.