Explaining Predictions of Non-Linear Classifiers in NLP
This work addresses the need for interpretability in NLP models, but it is incremental as it adapts an existing method (LRP) from image classification to a new domain.
The paper tackled the problem of explaining predictions of non-linear classifiers in NLP by applying layer-wise relevance propagation (LRP) to a CNN for topic categorization, showing that LRP effectively identifies relevant words and validates its suitability through experiments like word deleting and PCA analysis.
Layer-wise relevance propagation (LRP) is a recently proposed technique for explaining predictions of complex non-linear classifiers in terms of input variables. In this paper, we apply LRP for the first time to natural language processing (NLP). More precisely, we use it to explain the predictions of a convolutional neural network (CNN) trained on a topic categorization task. Our analysis highlights which words are relevant for a specific prediction of the CNN. We compare our technique to standard sensitivity analysis, both qualitatively and quantitatively, using a "word deleting" perturbation experiment, a PCA analysis, and various visualizations. All experiments validate the suitability of LRP for explaining the CNN predictions, which is also in line with results reported in recent image classification studies.