Learning Operations on a Stack with Neural Turing Machines
This addresses the challenge of storing information over long periods for sequence modeling tasks, but it is incremental as it applies an existing method to a specific problem.
The paper tackled the problem of long-term dependencies in sequence learning by testing Neural Turing Machines (NTMs) on recognizing well-balanced parentheses strings, showing that NTMs can emulate a stack and generalize to much longer sequences.
Multiple extensions of Recurrent Neural Networks (RNNs) have been proposed recently to address the difficulty of storing information over long time periods. In this paper, we experiment with the capacity of Neural Turing Machines (NTMs) to deal with these long-term dependencies on well-balanced strings of parentheses. We show that not only does the NTM emulate a stack with its heads and learn an algorithm to recognize such words, but it is also capable of strongly generalizing to much longer sequences.