Parallel Attention Mechanisms in Neural Machine Translation
This work addresses training efficiency and performance in machine translation, offering an incremental improvement over existing attention-based architectures.
The paper tackled the problem of reducing sequential operations in neural machine translation by proposing parallel attention mechanisms, which decreased training time and improved BLEU scores, establishing a new state of the art on English to German and English to French tasks.
Recent papers in neural machine translation have proposed the strict use of attention mechanisms over previous standards such as recurrent and convolutional neural networks (RNNs and CNNs). We propose that by running traditionally stacked encoding branches from encoder-decoder attention- focused architectures in parallel, that even more sequential operations can be removed from the model and thereby decrease training time. In particular, we modify the recently published attention-based architecture called Transformer by Google, by replacing sequential attention modules with parallel ones, reducing the amount of training time and substantially improving BLEU scores at the same time. Experiments over the English to German and English to French translation tasks show that our model establishes a new state of the art.