S3T: Self-Supervised Pre-training with Swin Transformer for Music Classification
This work addresses music classification for researchers and practitioners by improving representation learning from unlabeled data, though it is incremental as it adapts existing techniques to a new domain.
The paper tackled music classification by proposing S3T, a self-supervised pre-training method using Swin Transformer, which outperformed previous self-supervised methods by 12.5% top-1 accuracy and 4.8% PR-AUC on genre classification and tagging tasks, and surpassed supervised state-of-the-art methods.
In this paper, we propose S3T, a self-supervised pre-training method with Swin Transformer for music classification, aiming to learn meaningful music representations from massive easily accessible unlabeled music data. S3T introduces a momentum-based paradigm, MoCo, with Swin Transformer as its feature extractor to music time-frequency domain. For better music representations learning, S3T contributes a music data augmentation pipeline and two specially designed pre-processors. To our knowledge, S3T is the first method combining the Swin Transformer with a self-supervised learning method for music classification. We evaluate S3T on music genre classification and music tagging tasks with linear classifiers trained on learned representations. Experimental results show that S3T outperforms the previous self-supervised method (CLMR) by 12.5 percents top-1 accuracy and 4.8 percents PR-AUC on two tasks respectively, and also surpasses the task-specific state-of-the-art supervised methods. Besides, S3T shows advances in label efficiency using only 10% labeled data exceeding CLMR on both tasks with 100% labeled data.