An Attentive Survey of Attention Models
This is an incremental survey that organizes existing knowledge to guide practitioners in applying attention models across various domains.
The paper provides a structured survey and taxonomy of attention models in neural networks, reviewing their developments, architectures, applications, and impact on interpretability.
Attention Model has now become an important concept in neural networks that has been researched within diverse application domains. This survey provides a structured and comprehensive overview of the developments in modeling attention. In particular, we propose a taxonomy which groups existing techniques into coherent categories. We review salient neural architectures in which attention has been incorporated, and discuss applications in which modeling attention has shown a significant impact. We also describe how attention has been used to improve the interpretability of neural networks. Finally, we discuss some future research directions in attention. We hope this survey will provide a succinct introduction to attention models and guide practitioners while developing approaches for their applications.