Neural Attention Models in Deep Learning: Survey and Taxonomy
This work provides a structured understanding and taxonomy for researchers in machine learning and AI, but it is incremental as it synthesizes existing knowledge rather than introducing new methods.
The authors tackled the problem of organizing and analyzing the rapidly growing field of neural attention models in deep learning by conducting a survey and proposing a taxonomy based on theoretical aspects from psychology and neuroscience, resulting in a critical analysis of 51 main models from over 650 papers and highlighting unexplored theoretical issues.
Attention is a state of arousal capable of dealing with limited processing bottlenecks in human beings by focusing selectively on one piece of information while ignoring other perceptible information. For decades, concepts and functions of attention have been studied in philosophy, psychology, neuroscience, and computing. Currently, this property has been widely explored in deep neural networks. Many different neural attention models are now available and have been a very active research area over the past six years. From the theoretical standpoint of attention, this survey provides a critical analysis of major neural attention models. Here we propose a taxonomy that corroborates with theoretical aspects that predate Deep Learning. Our taxonomy provides an organizational structure that asks new questions and structures the understanding of existing attentional mechanisms. In particular, 17 criteria derived from psychology and neuroscience classic studies are formulated for qualitative comparison and critical analysis on the 51 main models found on a set of more than 650 papers analyzed. Also, we highlight several theoretical issues that have not yet been explored, including discussions about biological plausibility, highlight current research trends, and provide insights for the future.