InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective
This addresses robustness issues in NLP models for users relying on secure language processing, though it is incremental as it builds on existing adversarial training methods.
The paper tackles the vulnerability of BERT-based models to textual adversarial attacks by proposing InfoBERT, a framework with mutual-information-based regularizers, achieving state-of-the-art robust accuracy on adversarial datasets for NLI and QA tasks.
Large-scale language models such as BERT have achieved state-of-the-art performance across a wide range of NLP tasks. Recent studies, however, show that such BERT-based models are vulnerable facing the threats of textual adversarial attacks. We aim to address this problem from an information-theoretic perspective, and propose InfoBERT, a novel learning framework for robust fine-tuning of pre-trained language models. InfoBERT contains two mutual-information-based regularizers for model training: (i) an Information Bottleneck regularizer, which suppresses noisy mutual information between the input and the feature representation; and (ii) a Robust Feature regularizer, which increases the mutual information between local robust features and global features. We provide a principled way to theoretically analyze and improve the robustness of representation learning for language models in both standard and adversarial training. Extensive experiments demonstrate that InfoBERT achieves state-of-the-art robust accuracy over several adversarial datasets on Natural Language Inference (NLI) and Question Answering (QA) tasks. Our code is available at https://github.com/AI-secure/InfoBERT.