Spectrograms Are Sequences of Patches
This work addresses the lack of self-supervised methods in music processing, though it is incremental as it adapts existing techniques from NLP and CV to audio.
The authors tackled the problem of self-supervised pre-training for music by treating spectrograms as sequences of patches, achieving competitive results on downstream tasks compared to other audio representation models.
Self-supervised pre-training models have been used successfully in several machine learning domains. However, only a tiny amount of work is related to music. In our work, we treat a spectrogram of music as a series of patches and design a self-supervised model that captures the features of these sequential patches: Patchifier, which makes good use of self-supervised learning methods from both NLP and CV domains. We do not use labeled data for the pre-training process, only a subset of the MTAT dataset containing 16k music clips. After pre-training, we apply the model to several downstream tasks. Our model achieves a considerably acceptable result compared to other audio representation models. Meanwhile, our work demonstrates that it makes sense to consider audio as a series of patch segments.