Automatic Rule Extraction from Long Short Term Memory Networks
This addresses the black-box nature of deep learning models for researchers and practitioners in natural language processing, though it is incremental as it builds on existing LSTM interpretability efforts.
The paper tackles the problem of interpreting Long Short Term Memory networks (LSTMs) by developing a method to extract important input patterns, such as phrases, from state-of-the-art LSTMs used in sentiment analysis and question answering, enabling the construction of a rule-based classifier that approximates the LSTM's output.
Although deep learning models have proven effective at solving problems in natural language processing, the mechanism by which they come to their conclusions is often unclear. As a result, these models are generally treated as black boxes, yielding no insight of the underlying learned patterns. In this paper we consider Long Short Term Memory networks (LSTMs) and demonstrate a new approach for tracking the importance of a given input to the LSTM for a given output. By identifying consistently important patterns of words, we are able to distill state of the art LSTMs on sentiment analysis and question answering into a set of representative phrases. This representation is then quantitatively validated by using the extracted phrases to construct a simple, rule-based classifier which approximates the output of the LSTM.