Memory-Augmented Recurrent Neural Networks Can Learn Generalized Dyck Languages
This addresses the challenge of modeling hierarchical structures in language for AI and computational linguistics, representing an incremental advance in neural network capabilities.
The paper tackled the problem of neural networks recognizing hierarchical languages by introducing memory-augmented RNNs, achieving the first demonstration of learning generalized Dyck languages with up to six parenthesis-pairs, along with palindrome languages and string-reversal tasks.
We introduce three memory-augmented Recurrent Neural Networks (MARNNs) and explore their capabilities on a series of simple language modeling tasks whose solutions require stack-based mechanisms. We provide the first demonstration of neural networks recognizing the generalized Dyck languages, which express the core of what it means to be a language with hierarchical structure. Our memory-augmented architectures are easy to train in an end-to-end fashion and can learn the Dyck languages over as many as six parenthesis-pairs, in addition to two deterministic palindrome languages and the string-reversal transduction task, by emulating pushdown automata. Our experiments highlight the increased modeling capacity of memory-augmented models over simple RNNs, while inflecting our understanding of the limitations of these models.