The Unstoppable Rise of Computational Linguistics in Deep Learning
It provides a conceptual perspective on language's role in neural network development, which is incremental for researchers in computational linguistics and deep learning.
The paper traces the history of neural networks in natural language understanding, arguing that Transformers are induced-structure models rather than sequence models, and predicts future challenges in deep learning architectures for this domain.
In this paper, we trace the history of neural networks applied to natural language understanding tasks, and identify key contributions which the nature of language has made to the development of neural network architectures. We focus on the importance of variable binding and its instantiation in attention-based models, and argue that Transformer is not a sequence model but an induced-structure model. This perspective leads to predictions of the challenges facing research in deep learning architectures for natural language understanding.