Exploring Self-Supervised Contrastive Learning of Spatial Sound Event Representation
This work addresses spatial audio processing for applications like sound localization and event detection, presenting an incremental improvement through novel augmentations in a self-supervised approach.
The paper tackles the problem of learning spatial sound event representations from unlabeled audio by proposing MC-SimCLR, a multi-channel contrastive learning framework with multi-level augmentations, which improves event classification accuracy and reduces localization error compared to supervised models.
In this study, we present a simple multi-channel framework for contrastive learning (MC-SimCLR) to encode 'what' and 'where' of spatial audios. MC-SimCLR learns joint spectral and spatial representations from unlabeled spatial audios, thereby enhancing both event classification and sound localization in downstream tasks. At its core, we propose a multi-level data augmentation pipeline that augments different levels of audio features, including waveforms, Mel spectrograms, and generalized cross-correlation (GCC) features. In addition, we introduce simple yet effective channel-wise augmentation methods to randomly swap the order of the microphones and mask Mel and GCC channels. By using these augmentations, we find that linear layers on top of the learned representation significantly outperform supervised models in terms of both event classification accuracy and localization error. We also perform a comprehensive analysis of the effect of each augmentation method and a comparison of the fine-tuning performance using different amounts of labeled data.