Memory Time Span in LSTMs for Multi-Speaker Source Separation
This work provides insights into LSTM behavior for speech separation, which is incremental as it builds on existing deep learning methods to better understand network internals.
The paper analyzed how LSTMs handle temporal dependencies by measuring their memory time span using a controlled state leakage technique, applied to multi-speaker source separation, revealing both long-term (speaker characterization) and short-term (phone-size formant tracks) effects.
With deep learning approaches becoming state-of-the-art in many speech (as well as non-speech) related machine learning tasks, efforts are being taken to delve into the neural networks which are often considered as a black box. In this paper it is analyzed how recurrent neural network (RNNs) cope with temporal dependencies by determining the relevant memory time span in a long short-term memory (LSTM) cell. This is done by leaking the state variable with a controlled lifetime and evaluating the task performance. This technique can be used for any task to estimate the time span the LSTM exploits in that specific scenario. The focus in this paper is on the task of separating speakers from overlapping speech. We discern two effects: A long term effect, probably due to speaker characterization and a short term effect, probably exploiting phone-size formant tracks.