Learning Self-Supervised Audio-Visual Representations for Sound Recommendations
This work addresses the problem of sound recommendation for visual scenes, which is incremental as it builds on existing self-supervised and attention-based methods.
The paper tackled the problem of learning audio-visual representations from unlabeled videos for sound recommendations, achieving an 18% improvement in correlation accuracy and a 10% improvement in recommendation accuracy on the VGG-Sound dataset.
We propose a novel self-supervised approach for learning audio and visual representations from unlabeled videos, based on their correspondence. The approach uses an attention mechanism to learn the relative importance of convolutional features extracted at different resolutions from the audio and visual streams and uses the attention features to encode the audio and visual input based on their correspondence. We evaluated the representations learned by the model to classify audio-visual correlation as well as to recommend sound effects for visual scenes. Our results show that the representations generated by the attention model improves the correlation accuracy compared to the baseline, by 18% and the recommendation accuracy by 10% for VGG-Sound, which is a public video dataset. Additionally, audio-visual representations learned by training the attention model with cross-modal contrastive learning further improves the recommendation performance, based on our evaluation using VGG-Sound and a more challenging dataset consisting of gameplay video recordings.