Memory-Augmented Neural Networks for Machine Translation
This work addresses the limited real-world application of MANNs for machine translation, but it is incremental as it shows no significant improvement over existing attentional encoder-decoders.
The paper tackled the application of memory-augmented neural networks (MANNs) to machine translation, evaluating models like Neural Turing Machines and proposing extensions to attentional encoder-decoders, but found they performed similarly or worse (0.3-1.9 lower BLEU score) compared to standard methods on Vietnamese and Romanian to English tasks.
Memory-augmented neural networks (MANNs) have been shown to outperform other recurrent neural network architectures on a series of artificial sequence learning tasks, yet they have had limited application to real-world tasks. We evaluate direct application of Neural Turing Machines (NTM) and Differentiable Neural Computers (DNC) to machine translation. We further propose and evaluate two models which extend the attentional encoder-decoder with capabilities inspired by memory augmented neural networks. We evaluate our proposed models on IWSLT Vietnamese to English and ACL Romanian to English datasets. Our proposed models and the memory augmented neural networks perform similarly to the attentional encoder-decoder on the Vietnamese to English translation task while have a 0.3-1.9 lower BLEU score for the Romanian to English task. Interestingly, our analysis shows that despite being equipped with additional flexibility and being randomly initialized memory augmented neural networks learn an algorithm for machine translation almost identical to the attentional encoder-decoder.