Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures
This work addresses the theoretical understanding of neural machine translation architectures for researchers, showing that performance gains may be due to semantic feature extraction rather than long-range dependency modeling.
The paper tackled the problem of why non-recurrent architectures like CNNs and self-attention outperform RNNs in neural machine translation, finding that self-attentional networks do not outperform RNNs in modeling long-range dependencies but perform distinctly better on word sense disambiguation.
Recently, non-recurrent architectures (convolutional, self-attentional) have outperformed RNNs in neural machine translation. CNNs and self-attentional networks can connect distant words via shorter network paths than RNNs, and it has been speculated that this improves their ability to model long-range dependencies. However, this theoretical argument has not been tested empirically, nor have alternative explanations for their strong performance been explored in-depth. We hypothesize that the strong performance of CNNs and self-attentional networks could also be due to their ability to extract semantic features from the source text, and we evaluate RNNs, CNNs and self-attention networks on two tasks: subject-verb agreement (where capturing long-range dependencies is required) and word sense disambiguation (where semantic feature extraction is required). Our experimental results show that: 1) self-attentional networks and CNNs do not outperform RNNs in modeling subject-verb agreement over long distances; 2) self-attentional networks perform distinctly better than RNNs and CNNs on word sense disambiguation.