Interpretable Representation Learning for Speech and Audio Signals Based on Relevance Weighting
This work addresses the problem of interpretability in speech and audio processing for researchers and practitioners, offering a method that enhances performance while providing insights into the learned representations, though it is incremental as it builds on existing neural network architectures.
The authors tackled the challenge of learning interpretable representations from raw speech and audio signals by proposing a relevance weighting scheme integrated into a convolutional neural network, which improved performance on speech recognition and sound classification tasks across multiple datasets.
The learning of interpretable representations from raw data presents significant challenges for time series data like speech. In this work, we propose a relevance weighting scheme that allows the interpretation of the speech representations during the forward propagation of the model itself. The relevance weighting is achieved using a sub-network approach that performs the task of feature selection. A relevance sub-network, applied on the output of first layer of a convolutional neural network model operating on raw speech signals, acts as an acoustic filterbank (FB) layer with relevance weighting. A similar relevance sub-network applied on the second convolutional layer performs modulation filterbank learning with relevance weighting. The full acoustic model consisting of relevance sub-networks, convolutional layers and feed-forward layers is trained for a speech recognition task on noisy and reverberant speech in the Aurora-4, CHiME-3 and VOiCES datasets. The proposed representation learning framework is also applied for the task of sound classification in the UrbanSound8K dataset. A detailed analysis of the relevance weights learned by the model reveals that the relevance weights capture information regarding the underlying speech/audio content. In addition, speech recognition and sound classification experiments reveal that the incorporation of relevance weighting in the neural network architecture improves the performance significantly.