Spatio-Temporal Crop Aggregation for Video Representation Learning
This work addresses the challenge of efficient video representation learning for computer vision applications, offering incremental improvements in scalability and transfer learning.
The paper tackles the problem of learning scalable video representations by proposing SCALE, a method that aggregates spatio-temporal crops and uses a self-supervised masked prediction objective, achieving state-of-the-art performance on action classification and video understanding datasets.
We propose Spatio-temporal Crop Aggregation for video representation LEarning (SCALE), a novel method that enjoys high scalability at both training and inference time. Our model builds long-range video features by learning from sets of video clip-level features extracted with a pre-trained backbone. To train the model, we propose a self-supervised objective consisting of masked clip feature prediction. We apply sparsity to both the input, by extracting a random set of video clips, and to the loss function, by only reconstructing the sparse inputs. Moreover, we use dimensionality reduction by working in the latent space of a pre-trained backbone applied to single video clips. These techniques make our method not only extremely efficient to train but also highly effective in transfer learning. We demonstrate that our video representation yields state-of-the-art performance with linear, non-linear, and KNN probing on common action classification and video understanding datasets.