Rethinking Self-Attention: Towards Interpretability in Neural Parsing
This work addresses interpretability issues in NLP parsing for researchers and practitioners, though it is incremental as it builds on existing self-attention methods.
The authors tackled the problem of poor interpretability in self-attention mechanisms for neural parsing by introducing a Label Attention Layer, where attention heads represent labels, and achieved new state-of-the-art results on constituency and dependency parsing tasks on the Penn Treebank and Chinese Treebank.
Attention mechanisms have improved the performance of NLP tasks while allowing models to remain explainable. Self-attention is currently widely used, however interpretability is difficult due to the numerous attention distributions. Recent work has shown that model representations can benefit from label-specific information, while facilitating interpretation of predictions. We introduce the Label Attention Layer: a new form of self-attention where attention heads represent labels. We test our novel layer by running constituency and dependency parsing experiments and show our new model obtains new state-of-the-art results for both tasks on both the Penn Treebank (PTB) and Chinese Treebank. Additionally, our model requires fewer self-attention layers compared to existing work. Finally, we find that the Label Attention heads learn relations between syntactic categories and show pathways to analyze errors.