Multi-target DoA Estimation with an Audio-visual Fusion Mechanism
This addresses multi-speaker localization challenges like noise and occlusions for audio-visual systems, but it is incremental as it builds on prior multi-modal approaches.
The paper tackles multi-speaker localization by proposing an audio-visual fusion mechanism with adaptive weighting and a video simulation method, resulting in improved performance for speaker DoA estimation.
Most of the prior studies in the spatial \ac{DoA} domain focus on a single modality. However, humans use auditory and visual senses to detect the presence of sound sources. With this motivation, we propose to use neural networks with audio and visual signals for multi-speaker localization. The use of heterogeneous sensors can provide complementary information to overcome uni-modal challenges, such as noise, reverberation, illumination variations, and occlusions. We attempt to address these issues by introducing an adaptive weighting mechanism for audio-visual fusion. We also propose a novel video simulation method that generates visual features from noisy target 3D annotations that are synchronized with acoustic features. Experimental results confirm that audio-visual fusion consistently improves the performance of speaker DoA estimation, while the adaptive weighting mechanism shows clear benefits.