Spectrotemporal Modulation: Efficient and Interpretable Feature Representation for Classifying Speech, Music, and Environmental Sounds
This work addresses the need for efficient and interpretable audio feature representations in machine listening, with potential applications in auditory sciences and cognitive computing, though it is incremental as it builds on existing signal processing methods.
The paper tackled the problem of computationally costly and uninterpretable representations in audio deep neural networks by proposing spectrotemporal modulation features, achieving classification performance comparable to pretrained audio DNNs on speech, music, and environmental sounds without pretraining.
Audio DNNs have demonstrated impressive performance on various machine listening tasks; however, most of their representations are computationally costly and uninterpretable, leaving room for optimization. Here, we propose a novel approach centered on spectrotemporal modulation (STM) features, a signal processing method that mimics the neurophysiological representation in the human auditory cortex. The classification performance of our STM-based model, without any pretraining, is comparable to that of pretrained audio DNNs across diverse naturalistic speech, music, and environmental sounds, which are essential categories for both human cognition and machine perception. These results show that STM is an efficient and interpretable feature representation for audio classification, advancing the development of machine listening and unlocking exciting new possibilities for basic understanding of speech and auditory sciences, as well as developing audio BCI and cognitive computing.