The Importance of Being Recurrent for Modeling Hierarchical Structure
This addresses the need for effective architectures in NLP, but it is incremental as it builds on prior comparisons without introducing new methods.
The paper tackled the problem of modeling hierarchical structure in natural language processing by comparing recurrent and non-recurrent neural networks, finding that recurrency is important for this purpose.
Recent work has shown that recurrent neural networks (RNNs) can implicitly capture and exploit hierarchical information when trained to solve common natural language processing tasks such as language modeling (Linzen et al., 2016) and neural machine translation (Shi et al., 2016). In contrast, the ability to model structured data with non-recurrent neural networks has received little attention despite their success in many NLP tasks (Gehring et al., 2017; Vaswani et al., 2017). In this work, we compare the two architectures---recurrent versus non-recurrent---with respect to their ability to model hierarchical structure and find that recurrency is indeed important for this purpose.