Sequential Attention: A Context-Aware Alignment Function for Machine Reading
This addresses reading comprehension accuracy for NLP applications, representing an incremental improvement over existing attention mechanisms.
The paper tackles the problem of machine reading comprehension by proposing a Sequential Attention layer that extends soft attention to consider surrounding word context, showing dramatic improvements over the Stanford Reader baseline and competitive performance with state-of-the-art models on Who did What and CNN datasets.
In this paper we propose a neural network model with a novel Sequential Attention layer that extends soft attention by assigning weights to words in an input sequence in a way that takes into account not just how well that word matches a query, but how well surrounding words match. We evaluate this approach on the task of reading comprehension (on the Who did What and CNN datasets) and show that it dramatically improves a strong baseline--the Stanford Reader--and is competitive with the state of the art.