Cross-modal Spectrum Transformation Network For Acoustic Scene classification
This work addresses generalization issues in acoustic scene classification for applications like audio analysis, but it is incremental as it builds on existing CNN and cross-modal methods.
The paper tackles the problem of poor generalization in acoustic scene classification by transforming log-mel spectrums into imagined visual features using a cross-modal network, resulting in improved performance on DCASE and ESC-50 datasets, especially for unseen environments.
Convolutional neural networks (CNNs) with log-mel spectrum features have shown promising results for acoustic scene classification tasks. However, the performance of these CNN based classifiers is still lacking as they do not generalise well for unknown environments. To address this issue, we introduce an acoustic spectrum transformation network where traditional log-mel spectrums are transformed into imagined visual features (IVF). The imagined visual features are learned by exploiting the relationship between audio and visual features present in video recordings. An auto-encoder is used to encode images as visual features and a transformation network learns how to generate imagined visual features from log-mel. Our model is trained on a large dataset of Youtube videos. We test our proposed method on the scene classification task of DCASE and ESC-50, where our method outperforms other spectrum features, especially for unseen environments.