Adversarial Self-Attention for Language Understanding
This addresses robustness and generalization issues in NLP models for researchers and practitioners, but it is incremental as it builds on existing self-attention mechanisms.
The paper tackles the problem of spurious features in Transformer-based language models impairing generalization and robustness by proposing an Adversarial Self-Attention mechanism that biases attentions to suppress reliance on specific features and encourage broader semantics, resulting in remarkable performance gains in pre-training and large margins in fine-tuning for generalization and robustness.
Deep neural models (e.g. Transformer) naturally learn spurious features, which create a ``shortcut'' between the labels and inputs, thus impairing the generalization and robustness. This paper advances the self-attention mechanism to its robust variant for Transformer-based pre-trained language models (e.g. BERT). We propose \textit{Adversarial Self-Attention} mechanism (ASA), which adversarially biases the attentions to effectively suppress the model reliance on features (e.g. specific keywords) and encourage its exploration of broader semantics. We conduct a comprehensive evaluation across a wide range of tasks for both pre-training and fine-tuning stages. For pre-training, ASA unfolds remarkable performance gains compared to naive training for longer steps. For fine-tuning, ASA-empowered models outweigh naive models by a large margin considering both generalization and robustness.