Guiding Attention for Self-Supervised Learning with Transformers
This work addresses a bottleneck in self-supervised learning for NLP, offering an incremental improvement with practical benefits for low-resource applications.
The paper tackled the problem of inefficient self-supervised learning with Transformers by proposing an auxiliary loss to guide attention heads, resulting in faster convergence and state-of-the-art performance in low-resource settings.
In this paper, we propose a simple and effective technique to allow for efficient self-supervised learning with bi-directional Transformers. Our approach is motivated by recent studies demonstrating that self-attention patterns in trained models contain a majority of non-linguistic regularities. We propose a computationally efficient auxiliary loss function to guide attention heads to conform to such patterns. Our method is agnostic to the actual pre-training objective and results in faster convergence of models as well as better performance on downstream tasks compared to the baselines, achieving state of the art results in low-resource settings. Surprisingly, we also find that linguistic properties of attention heads are not necessarily correlated with language modeling performance.