Controllable Augmentations for Video Representation Learning
This work addresses limitations in video representation learning for tasks like action recognition and retrieval, but it is incremental as it builds on existing contrastive learning methods.
The paper tackles the problem of bias towards static backgrounds and difficulty in establishing global temporal structures in self-supervised video representation learning by proposing a framework that uses local clips and global videos with controllable augmentations. It achieves superior performance on three video benchmarks for action recognition and video retrieval, capturing more accurate temporal dynamics.
This paper focuses on self-supervised video representation learning. Most existing approaches follow the contrastive learning pipeline to construct positive and negative pairs by sampling different clips. However, this formulation tends to bias to static background and have difficulty establishing global temporal structures. The major reason is that the positive pairs, i.e., different clips sampled from the same video, have limited temporal receptive field, and usually share similar background but differ in motions. To address these problems, we propose a framework to jointly utilize local clips and global videos to learn from detailed region-level correspondence as well as general long-term temporal relations. Based on a set of controllable augmentations, we achieve accurate appearance and motion pattern alignment through soft spatio-temporal region contrast. Our formulation is able to avoid the low-level redundancy shortcut by mutual information minimization to improve the generalization. We also introduce local-global temporal order dependency to further bridge the gap between clip-level and video-level representations for robust temporal modeling. Extensive experiments demonstrate that our framework is superior on three video benchmarks in action recognition and video retrieval, capturing more accurate temporal dynamics.