Analyzing and Interpreting Neural Networks for NLP: A Report on the First BlackboxNLP Workshop
This addresses the challenge of making neural models more transparent for researchers and practitioners in NLP, but it is incremental as it summarizes existing workshop contributions.
The paper reviews studies from the BlackboxNLP workshop that tackled the problem of interpreting neural networks in NLP, focusing on methods like input manipulation, decoding knowledge from representations, and architectural modifications to enhance explainability.
The EMNLP 2018 workshop BlackboxNLP was dedicated to resources and techniques specifically developed for analyzing and understanding the inner-workings and representations acquired by neural models of language. Approaches included: systematic manipulation of input to neural networks and investigating the impact on their performance, testing whether interpretable knowledge can be decoded from intermediate representations acquired by neural networks, proposing modifications to neural network architectures to make their knowledge state or generated output more explainable, and examining the performance of networks on simplified or formal languages. Here we review a number of representative studies in each category.