Exploiting Audio-Visual Consistency with Partial Supervision for Spatial Audio Generation
This addresses the problem of degraded audio experience in videos for users, but is incremental as it builds on existing audio spatialization methods with a new learning approach.
The paper tackles the problem of converting monaural videos to binaural audio to enhance user experience by exploiting audio-visual consistency, using a self-supervised learning strategy that reduces dependency on ground truth binaural data. Experiments on benchmark datasets confirm the framework's effectiveness in both semi-supervised and fully supervised scenarios.
Human perceives rich auditory experience with distinct sound heard by ears. Videos recorded with binaural audio particular simulate how human receives ambient sound. However, a large number of videos are with monaural audio only, which would degrade the user experience due to the lack of ambient information. To address this issue, we propose an audio spatialization framework to convert a monaural video into a binaural one exploiting the relationship across audio and visual components. By preserving the left-right consistency in both audio and visual modalities, our learning strategy can be viewed as a self-supervised learning technique, and alleviates the dependency on a large amount of video data with ground truth binaural audio data during training. Experiments on benchmark datasets confirm the effectiveness of our proposed framework in both semi-supervised and fully supervised scenarios, with ablation studies and visualization further support the use of our model for audio spatialization.