An Introductory Survey on Attention Mechanisms in NLP Problems
It offers an introductory overview for readers seeking basic knowledge on attention mechanisms in NLP, making it incremental as it compiles existing research without novel contributions.
This paper provides a survey of attention mechanisms in NLP, summarizing their applications across tasks like sentiment classification and question answering, but does not present new experimental results or concrete numbers.
First derived from human intuition, later adapted to machine translation for automatic token alignment, attention mechanism, a simple method that can be used for encoding sequence data based on the importance score each element is assigned, has been widely applied to and attained significant improvement in various tasks in natural language processing, including sentiment classification, text summarization, question answering, dependency parsing, etc. In this paper, we survey through recent works and conduct an introductory summary of the attention mechanism in different NLP problems, aiming to provide our readers with basic knowledge on this widely used method, discuss its different variants for different tasks, explore its association with other techniques in machine learning, and examine methods for evaluating its performance.