DropAttention: A Regularization Method for Fully-Connected Self-Attention Networks
This addresses overfitting in attention-based models like Transformers, which are widely used in NLP and other domains, but it is incremental as it adapts dropout to a specific layer.
The paper tackled the lack of a specific dropout method for fully-connected self-attention layers in Transformers by proposing DropAttention to regularize attention weights, resulting in improved performance and reduced overfitting across various tasks.
Variants dropout methods have been designed for the fully-connected layer, convolutional layer and recurrent layer in neural networks, and shown to be effective to avoid overfitting. As an appealing alternative to recurrent and convolutional layers, the fully-connected self-attention layer surprisingly lacks a specific dropout method. This paper explores the possibility of regularizing the attention weights in Transformers to prevent different contextualized feature vectors from co-adaption. Experiments on a wide range of tasks show that DropAttention can improve performance and reduce overfitting.