Deep Audio-Visual Learning: A Survey
It synthesizes existing research for researchers in multimodal AI, but is incremental as it does not introduce new methods or results.
This paper provides a comprehensive survey of recent developments in audio-visual learning, categorizing tasks into four subfields and discussing state-of-the-art methods and challenges.
Audio-visual learning, aimed at exploiting the relationship between audio and visual modalities, has drawn considerable attention since deep learning started to be used successfully. Researchers tend to leverage these two modalities either to improve the performance of previously considered single-modality tasks or to address new challenging problems. In this paper, we provide a comprehensive survey of recent audio-visual learning development. We divide the current audio-visual learning tasks into four different subfields: audio-visual separation and localization, audio-visual correspondence learning, audio-visual generation, and audio-visual representation learning. State-of-the-art methods as well as the remaining challenges of each subfield are further discussed. Finally, we summarize the commonly used datasets and performance metrics.