Learning Sound Localization Better From Semantically Similar Samples
This work addresses a specific bottleneck in audio-visual localization for researchers, offering an incremental improvement over existing contrastive learning approaches.
The paper tackles the problem of sound source localization in visual scenes by addressing the issue of semantically similar audio-visual pairs being incorrectly treated as negatives in contrastive learning. It proposes incorporating these 'hard positives' into the objective, achieving favorable performance on VGG-SS and SoundNet-Flickr test sets compared to state-of-the-art methods.
The objective of this work is to localize the sound sources in visual scenes. Existing audio-visual works employ contrastive learning by assigning corresponding audio-visual pairs from the same source as positives while randomly mismatched pairs as negatives. However, these negative pairs may contain semantically matched audio-visual information. Thus, these semantically correlated pairs, "hard positives", are mistakenly grouped as negatives. Our key contribution is showing that hard positives can give similar response maps to the corresponding pairs. Our approach incorporates these hard positives by adding their response maps into a contrastive learning objective directly. We demonstrate the effectiveness of our approach on VGG-SS and SoundNet-Flickr test sets, showing favorable performance to the state-of-the-art methods.