Learning to Transduce with Unbounded Memory
This work addresses the problem of improving transduction tasks like machine translation for AI researchers by introducing novel architectures, though it is incremental as it builds on existing deep RNNs.
The paper tackled the limited representational power of deep recurrent neural networks in natural language transduction by proposing new memory-based recurrent networks that mimic traditional data structures like stacks and queues, showing superior generalization performance and often learning the underlying generating algorithms in synthetic grammar experiments.
Recently, strong results have been demonstrated by Deep Recurrent Neural Networks on natural language transduction problems. In this paper we explore the representational power of these models using synthetic grammars designed to exhibit phenomena similar to those found in real transduction problems such as machine translation. These experiments lead us to propose new memory-based recurrent networks that implement continuously differentiable analogues of traditional data structures such as Stacks, Queues, and DeQues. We show that these architectures exhibit superior generalisation performance to Deep RNNs and are often able to learn the underlying generating algorithms in our transduction experiments.