Unitary Multi-Margin BERT for Robust Natural Language Processing
This addresses the vulnerability of mission-critical NLP systems to adversarial exploitation, offering a computationally efficient defense method that is incremental in improving robustness.
The paper tackles the problem of adversarial attacks on BERT models in NLP by introducing a novel defense method combining unitary weights and multi-margin loss, resulting in a significant boost in post-attack classification accuracies by 5.3% to 73.8% while maintaining competitive pre-attack performance.
Recent developments in adversarial attacks on deep learning leave many mission-critical natural language processing (NLP) systems at risk of exploitation. To address the lack of computationally efficient adversarial defense methods, this paper reports a novel, universal technique that drastically improves the robustness of Bidirectional Encoder Representations from Transformers (BERT) by combining the unitary weights with the multi-margin loss. We discover that the marriage of these two simple ideas amplifies the protection against malicious interference. Our model, the unitary multi-margin BERT (UniBERT), boosts post-attack classification accuracies significantly by 5.3% to 73.8% while maintaining competitive pre-attack accuracies. Furthermore, the pre-attack and post-attack accuracy tradeoff can be adjusted via a single scalar parameter to best fit the design requirements for the target applications.