IM-BERT: Enhancing Robustness of BERT through the Implicit Euler Method
This addresses the problem of adversarial robustness for NLP practitioners using BERT, offering an incremental improvement through a novel connection method.
The paper tackles the vulnerability of pre-trained language models like BERT to adversarial attacks, especially in low-resource settings, by proposing IM-BERT, which enhances robustness without extra parameters or adversarial training, resulting in an 8.3% performance improvement on the AdvGLUE dataset and 5.9% higher accuracy in low-resource scenarios.
Pre-trained Language Models (PLMs) have achieved remarkable performance on diverse NLP tasks through pre-training and fine-tuning. However, fine-tuning the model with a large number of parameters on limited downstream datasets often leads to vulnerability to adversarial attacks, causing overfitting of the model on standard datasets. To address these issues, we propose IM-BERT from the perspective of a dynamic system by conceptualizing a layer of BERT as a solution of Ordinary Differential Equations (ODEs). Under the situation of initial value perturbation, we analyze the numerical stability of two main numerical ODE solvers: the explicit and implicit Euler approaches. Based on these analyses, we introduce a numerically robust IM-connection incorporating BERT's layers. This strategy enhances the robustness of PLMs against adversarial attacks, even in low-resource scenarios, without introducing additional parameters or adversarial training strategies. Experimental results on the adversarial GLUE (AdvGLUE) dataset validate the robustness of IM-BERT under various conditions. Compared to the original BERT, IM-BERT exhibits a performance improvement of approximately 8.3\%p on the AdvGLUE dataset. Furthermore, in low-resource scenarios, IM-BERT outperforms BERT by achieving 5.9\%p higher accuracy.