Learning Audio-Visual Correlations from Variational Cross-Modal Generation
This work addresses the problem of learning audio-visual correlations for various downstream tasks, which is an incremental step for the multimedia research community.
This paper proposes a self-supervised Variational AutoEncoder (VAE) framework, called MS-VAE, to learn intrinsic audio-visual correlations through cross-modal generation. The learned representations achieve competitive performance in downstream tasks like audio-visual cross-modal localization and retrieval without labeled training data.
People can easily imagine the potential sound while seeing an event. This natural synchronization between audio and visual signals reveals their intrinsic correlations. To this end, we propose to learn the audio-visual correlations from the perspective of cross-modal generation in a self-supervised manner, the learned correlations can be then readily applied in multiple downstream tasks such as the audio-visual cross-modal localization and retrieval. We introduce a novel Variational AutoEncoder (VAE) framework that consists of Multiple encoders and a Shared decoder (MS-VAE) with an additional Wasserstein distance constraint to tackle the problem. Extensive experiments demonstrate that the optimized latent representation of the proposed MS-VAE can effectively learn the audio-visual correlations and can be readily applied in multiple audio-visual downstream tasks to achieve competitive performance even without any given label information during training.