Hierarchically Decoupled Spatial-Temporal Contrast for Self-supervised Video Representation Learning
This work provides an incremental improvement in self-supervised video representation learning for researchers and practitioners working on video understanding tasks.
This paper tackles self-supervised video representation learning by decoupling spatial and temporal feature learning and applying it hierarchically. The proposed Hierarchically Decoupled Spatial-Temporal Contrast (HDC) method achieves substantial improvements over direct spatial-temporal learning and competitive performance against state-of-the-art unsupervised methods on action recognition benchmarks like UCF101 and HMDB51.
We present a novel technique for self-supervised video representation learning by: (a) decoupling the learning objective into two contrastive subtasks respectively emphasizing spatial and temporal features, and (b) performing it hierarchically to encourage multi-scale understanding. Motivated by their effectiveness in supervised learning, we first introduce spatial-temporal feature learning decoupling and hierarchical learning to the context of unsupervised video learning. We show by experiments that augmentations can be manipulated as regularization to guide the network to learn desired semantics in contrastive learning, and we propose a way for the model to separately capture spatial and temporal features at multiple scales. We also introduce an approach to overcome the problem of divergent levels of instance invariance at different hierarchies by modeling the invariance as loss weights for objective re-weighting. Experiments on downstream action recognition benchmarks on UCF101 and HMDB51 show that our proposed Hierarchically Decoupled Spatial-Temporal Contrast (HDC) makes substantial improvements over directly learning spatial-temporal features as a whole and achieves competitive performance when compared with other state-of-the-art unsupervised methods. Code will be made available.