Frequency Selective Augmentation for Video Representation Learning
This addresses a specific bottleneck in video representation learning for computer vision applications, offering an incremental enhancement to existing self-supervised frameworks.
The paper tackles bias towards static information in self-supervised video representation learning by proposing FreqAug, a frequency-domain augmentation method that removes specific frequency components, resulting in consistent improvements across multiple downstream tasks like action recognition and temporal action localization.
Recent self-supervised video representation learning methods focus on maximizing the similarity between multiple augmented views from the same video and largely rely on the quality of generated views. However, most existing methods lack a mechanism to prevent representation learning from bias towards static information in the video. In this paper, we propose frequency augmentation (FreqAug), a spatio-temporal data augmentation method in the frequency domain for video representation learning. FreqAug stochastically removes specific frequency components from the video so that learned representation captures essential features more from the remaining information for various downstream tasks. Specifically, FreqAug pushes the model to focus more on dynamic features rather than static features in the video via dropping spatial or temporal low-frequency components. To verify the generality of the proposed method, we experiment with FreqAug on multiple self-supervised learning frameworks along with standard augmentations. Transferring the improved representation to five video action recognition and two temporal action localization downstream tasks shows consistent improvements over baselines.