Cross-modal supervised learning for better acoustic representations
This addresses the data labeling bottleneck for audio representation learning, though it appears incremental as it builds on existing CNN architectures with automatic labeling.
The authors tackled the challenge of obtaining large-scale human-labeled datasets for acoustic representation learning by using machine-generated labels from synchronized vision-audio data, achieving significant performance boosts over state-of-the-art results on three external audio classification benchmarks.
Obtaining large-scale human-labeled datasets to train acoustic representation models is a very challenging task. On the contrary, we can easily collect data with machine-generated labels. In this work, we propose to exploit machine-generated labels to learn better acoustic representations, based on the synchronization between vision and audio. Firstly, we collect a large-scale video dataset with 15 million samples, which totally last 16,320 hours. Each video is 3 to 5 seconds in length and annotated automatically by publicly available visual and audio classification models. Secondly, we train various classical convolutional neural networks (CNNs) including VGGish, ResNet 50 and Mobilenet v2. We also make several improvements to VGGish and achieve better results. Finally, we transfer our models on three external standard benchmarks for audio classification task, and achieve significant performance boost over the state-of-the-art results. Models and codes are available at: https://github.com/Deeperjia/vgg-like-audio-models.