Explaining and Interpreting LSTMs
This work addresses interpretability for users of sequential data models, but it is incremental as it extends an existing method to a specific architecture.
The authors tackled the problem of explaining predictions from LSTM neural networks by adapting the Layer-wise Relevance Propagation technique, resulting in a new propagation scheme and theoretical extension to handle LSTM's gated interactions.
While neural networks have acted as a strong unifying force in the design of modern AI systems, the neural network architectures themselves remain highly heterogeneous due to the variety of tasks to be solved. In this chapter, we explore how to adapt the Layer-wise Relevance Propagation (LRP) technique used for explaining the predictions of feed-forward networks to the LSTM architecture used for sequential data modeling and forecasting. The special accumulators and gated interactions present in the LSTM require both a new propagation scheme and an extension of the underlying theoretical framework to deliver faithful explanations.