Self-Attention with Structural Position Representations
This addresses the problem of improving position encoding in NLP models for machine translation, but it is incremental as it builds on existing self-attention networks.
The paper tackled the limitation of self-attention networks in encoding word positions by augmenting them with structural position representations based on dependency trees, resulting in consistent performance boosts on NIST Chinese-to-English and WMT14 English-to-German translation tasks.
Although self-attention networks (SANs) have advanced the state-of-the-art on various NLP tasks, one criticism of SANs is their ability of encoding positions of input words (Shaw et al., 2018). In this work, we propose to augment SANs with structural position representations to model the latent structure of the input sentence, which is complementary to the standard sequential positional representations. Specifically, we use dependency tree to represent the grammatical structure of a sentence, and propose two strategies to encode the positional relationships among words in the dependency tree. Experimental results on NIST Chinese-to-English and WMT14 English-to-German translation tasks show that the proposed approach consistently boosts performance over both the absolute and relative sequential position representations.