Recursive Visual Sound Separation Using Minus-Plus Net
This addresses the challenge of accurate audio-visual separation for applications like multimedia analysis, though it appears incremental as it builds on existing visual sound separation methods.
The paper tackles the problem of separating mixed sounds from multiple sources using visual cues, proposing the MinusPlus Network (MP-Net) that recursively separates sounds by energy order, achieving state-of-the-art results on two large-scale datasets.
Sounds provide rich semantics, complementary to visual data, for many tasks. However, in practice, sounds from multiple sources are often mixed together. In this paper we propose a novel framework, referred to as MinusPlus Network (MP-Net), for the task of visual sound separation. MP-Net separates sounds recursively in the order of average energy, removing the separated sound from the mixture at the end of each prediction, until the mixture becomes empty or contains only noise. In this way, MP-Net could be applied to sound mixtures with arbitrary numbers and types of sounds. Moreover, while MP-Net keeps removing sounds with large energy from the mixture, sounds with small energy could emerge and become clearer, so that the separation is more accurate. Compared to previous methods, MP-Net obtains state-of-the-art results on two large scale datasets, across mixtures with different types and numbers of sounds.