Generalized Attention Mechanism and Relative Position for Transformer
This work addresses a domain-specific issue for NLP researchers by offering incremental improvements to attention mechanisms in Transformers.
The authors tackled the problem of improving Transformer models by proposing a generalized attention mechanism (GAM) and a new relative position representation, resulting in a framework that can handle cases where input sequence elements are at random locations in datasets.
In this paper, we propose generalized attention mechanism (GAM) by first suggesting a new interpretation for self-attention mechanism of Vaswani et al. . Following the interpretation, we provide description for different variants of attention mechanism which together form GAM. Further, we propose a new relative position representation within the framework of GAM. This representation can be easily utilized for cases in which elements next to each other in input sequence can be at random locations in actual dataset/corpus.