The Impact of Spatiotemporal Augmentations on Self-Supervised Audiovisual Representation Learning
This work addresses the challenge of integrating auditory and visual perception for researchers in self-supervised learning, though it is incremental as it builds on existing contrastive methods with specific augmentations.
The paper tackles the problem of learning effective audiovisual representations from unlabeled videos by investigating spatiotemporal augmentations in a contrastive framework, resulting in a 6.5% gain in linear classifier performance on the AVE dataset compared to models using only sampling-based augmentations.
Contrastive learning of auditory and visual perception has been extremely successful when investigated individually. However, there are still major questions on how we could integrate principles learned from both domains to attain effective audiovisual representations. In this paper, we present a contrastive framework to learn audiovisual representations from unlabeled videos. The type and strength of augmentations utilized during self-supervised pre-training play a crucial role for contrastive frameworks to work sufficiently. Hence, we extensively investigate composition of temporal augmentations suitable for learning audiovisual representations; we find lossy spatio-temporal transformations that do not corrupt the temporal coherency of videos are the most effective. Furthermore, we show that the effectiveness of these transformations scales with higher temporal resolution and stronger transformation intensity. Compared to self-supervised models pre-trained on only sampling-based temporal augmentation, self-supervised models pre-trained with our temporal augmentations lead to approximately 6.5% gain on linear classifier performance on AVE dataset. Lastly, we show that despite their simplicity, our proposed transformations work well across self-supervised learning frameworks (SimSiam, MoCoV3, etc), and benchmark audiovisual dataset (AVE).