FullLoRA: Efficiently Boosting the Robustness of Pretrained Vision Transformers
This addresses security concerns for users of ViT models by providing a parameter-efficient method to boost robustness, though it is incremental as it builds on existing LoRA techniques.
The paper tackles the problem of enhancing adversarial robustness in pretrained Vision Transformers (ViTs) by proposing the FullLoRA framework, which integrates learnable LNLoRA modules into all key components while keeping the pretrained model frozen, achieving comparable robustness to full finetuning with only about 5% of the learnable parameters.
In recent years, the Vision Transformer (ViT) model has gradually become mainstream in various computer vision tasks, and the robustness of the model has received increasing attention. However, existing large models tend to prioritize performance during training, potentially neglecting the robustness, which may lead to serious security concerns. In this paper, we establish a new challenge: exploring how to use a small number of additional parameters for adversarial finetuning to quickly and effectively enhance the adversarial robustness of a standardly trained model. To address this challenge, we develop novel LNLoRA module, incorporating a learnable layer normalization before the conventional LoRA module, which helps mitigate magnitude differences in parameters between the adversarial and standard training paradigms. Furthermore, we propose the FullLoRA framework by integrating the learnable LNLoRA modules into all key components of ViT-based models while keeping the pretrained model frozen, which can significantly improve the model robustness via adversarial finetuning in a parameter-efficient manner. Extensive experiments on several datasets demonstrate the superiority of our proposed FullLoRA framework. It achieves comparable robustness with full finetuning while only requiring about 5\% of the learnable parameters. This also effectively addresses concerns regarding extra model storage space and enormous training time caused by adversarial finetuning.