Linguistic Knowledge as Memory for Recurrent Neural Networks
This addresses the challenge of long-term dependencies in NLP tasks, offering a novel approach that improves performance on coreference and text comprehension, though it is incremental in combining existing ideas with linguistic knowledge.
The paper tackles the problem of modeling long-term dependencies in recurrent neural networks by using external linguistic knowledge as explicit memory, achieving new state-of-the-art results on text comprehension benchmarks such as CNN, bAbi, and LAMBADA, including solving 15 out of 20 bAbi QA tasks with only 1000 training examples per task.
Training recurrent neural networks to model long term dependencies is difficult. Hence, we propose to use external linguistic knowledge as an explicit signal to inform the model which memories it should utilize. Specifically, external knowledge is used to augment a sequence with typed edges between arbitrarily distant elements, and the resulting graph is decomposed into directed acyclic subgraphs. We introduce a model that encodes such graphs as explicit memory in recurrent neural networks, and use it to model coreference relations in text. We apply our model to several text comprehension tasks and achieve new state-of-the-art results on all considered benchmarks, including CNN, bAbi, and LAMBADA. On the bAbi QA tasks, our model solves 15 out of the 20 tasks with only 1000 training examples per task. Analysis of the learned representations further demonstrates the ability of our model to encode fine-grained entity information across a document.