Analysis of memory in LSTM-RNNs for source separation
This provides insights into memory mechanisms in RNNs for researchers in speech processing, though it is incremental as it analyzes existing methods.
The paper tackled the problem of understanding what information LSTM-RNNs store and for how long in speech processing, finding that short-term linguistic processes affect performance within 100 milliseconds, while speaker characteristics are retained beyond 400 milliseconds.
Long short-term memory recurrent neural networks (LSTM-RNNs) are considered state-of-the art in many speech processing tasks. The recurrence in the network, in principle, allows any input to be remembered for an indefinite time, a feature very useful for sequential data like speech. However, very little is known about which information is actually stored in the LSTM and for how long. We address this problem by using a memory reset approach which allows us to evaluate network performance depending on the allowed memory time span. We apply this approach to the task of multi-speaker source separation, but it can be used for any task using RNNs. We find a strong performance effect of short-term (shorter than 100 milliseconds) linguistic processes. Only speaker characteristics are kept in the memory for longer than 400 milliseconds. Furthermore, we confirm that performance-wise it is sufficient to implement longer memory in deeper layers. Finally, in a bidirectional model, the backward models contributes slightly more to the separation performance than the forward model.