Mask Attention Networks: Rethinking and Strengthen Transformer
This is an incremental improvement for natural language processing tasks, enhancing text representation learning in Transformers.
The paper tackles the limitation of static mask matrices in Transformer's Self-Attention and Feed-Forward Networks by introducing a dynamic mask attention network (DMAN) with a learnable mask matrix for adaptive localness modeling, and experiments show it outperforms the original Transformer on tasks like neural machine translation and text summarization.
Transformer is an attention-based neural network, which consists of two sublayers, namely, Self-Attention Network (SAN) and Feed-Forward Network (FFN). Existing research explores to enhance the two sublayers separately to improve the capability of Transformer for text representation. In this paper, we present a novel understanding of SAN and FFN as Mask Attention Networks (MANs) and show that they are two special cases of MANs with static mask matrices. However, their static mask matrices limit the capability for localness modeling in text representation learning. We therefore introduce a new layer named dynamic mask attention network (DMAN) with a learnable mask matrix which is able to model localness adaptively. To incorporate advantages of DMAN, SAN, and FFN, we propose a sequential layered structure to combine the three types of layers. Extensive experiments on various tasks, including neural machine translation and text summarization demonstrate that our model outperforms the original Transformer.