The Impact of Quantization on the Robustness of Transformer-based Text Classifiers
This addresses the problem of adversarial vulnerabilities in NLP models for practitioners, offering a computationally efficient robustness improvement, though it is incremental as it applies an existing technique to a new aspect.
The paper investigates how quantization affects the robustness of Transformer-based text classifiers against adversarial attacks, finding that it improves adversarial accuracy by an average of 18.68% and outperforms adversarial training by 18.80% without extra computational cost.
Transformer-based models have made remarkable advancements in various NLP areas. Nevertheless, these models often exhibit vulnerabilities when confronted with adversarial attacks. In this paper, we explore the effect of quantization on the robustness of Transformer-based models. Quantization usually involves mapping a high-precision real number to a lower-precision value, aiming at reducing the size of the model at hand. To the best of our knowledge, this work is the first application of quantization on the robustness of NLP models. In our experiments, we evaluate the impact of quantization on BERT and DistilBERT models in text classification using SST-2, Emotion, and MR datasets. We also evaluate the performance of these models against TextFooler, PWWS, and PSO adversarial attacks. Our findings show that quantization significantly improves (by an average of 18.68%) the adversarial accuracy of the models. Furthermore, we compare the effect of quantization versus that of the adversarial training approach on robustness. Our experiments indicate that quantization increases the robustness of the model by 18.80% on average compared to adversarial training without imposing any extra computational overhead during training. Therefore, our results highlight the effectiveness of quantization in improving the robustness of NLP models.