SLICER: Learning universal audio representations using low-resource self-supervised pre-training
This addresses the need for efficient audio representation learning in low-resource settings, benefiting speech and audio classification tasks, though it appears incremental in combining existing paradigms.
The paper tackles the problem of learning universal audio representations with limited labeled data by proposing SLICER, a self-supervised pre-training method that combines clustering and contrastive learning with a novel augmentation technique. It achieves state-of-the-art results on the LAPE Benchmark, outperforming prior approaches pre-trained on 10× larger datasets.
We present a new Self-Supervised Learning (SSL) approach to pre-train encoders on unlabeled audio data that reduces the need for large amounts of labeled data for audio and speech classification. Our primary aim is to learn audio representations that can generalize across a large variety of speech and non-speech tasks in a low-resource un-labeled audio pre-training setting. Inspired by the recent success of clustering and contrasting learning paradigms for SSL-based speech representation learning, we propose SLICER (Symmetrical Learning of Instance and Cluster-level Efficient Representations), which brings together the best of both clustering and contrasting learning paradigms. We use a symmetric loss between latent representations from student and teacher encoders and simultaneously solve instance and cluster-level contrastive learning tasks. We obtain cluster representations online by just projecting the input spectrogram into an output subspace with dimensions equal to the number of clusters. In addition, we propose a novel mel-spectrogram augmentation procedure, k-mix, based on mixup, which does not require labels and aids unsupervised representation learning for audio. Overall, SLICER achieves state-of-the-art results on the LAPE Benchmark \cite{9868132}, significantly outperforming DeLoRes-M and other prior approaches, which are pre-trained on $10\times$ larger of unsupervised data. We will make all our codes available on GitHub.