Audio-Visual Scene Analysis with Self-Supervised Multisensory Features
This work addresses the challenge of integrating audio and visual data for scene analysis, offering incremental improvements in multimodal learning for tasks like source separation and recognition.
The paper tackles the problem of joint modeling of visual and audio signals in videos by learning a fused multisensory representation through self-supervised training to predict temporal alignment, achieving applications such as sound source localization, audio-visual action recognition, and on/off-screen audio source separation.
The thud of a bouncing ball, the onset of speech as lips open -- when visual and audio events occur together, it suggests that there might be a common, underlying event that produced both signals. In this paper, we argue that the visual and audio components of a video signal should be modeled jointly using a fused multisensory representation. We propose to learn such a representation in a self-supervised way, by training a neural network to predict whether video frames and audio are temporally aligned. We use this learned representation for three applications: (a) sound source localization, i.e. visualizing the source of sound in a video; (b) audio-visual action recognition; and (c) on/off-screen audio source separation, e.g. removing the off-screen translator's voice from a foreign official's speech. Code, models, and video results are available on our webpage: http://andrewowens.com/multisensory