AdaGAT: Adaptive Guidance Adversarial Training for the Robustness of Deep Neural Networks
This work addresses the problem of improving robustness in lightweight student models for adversarial attacks, though it is incremental as it builds on existing adversarial distillation methods.
The paper tackles the challenge of maintaining an optimal teacher model during adversarial distillation for robust deep neural networks, proposing AdaGAT to dynamically adjust the guide model's training state, which enhances the target model's robustness across adversarial attacks on datasets like CIFAR-10, CIFAR-100, and TinyImageNet.
Adversarial distillation (AD) is a knowledge distillation technique that facilitates the transfer of robustness from teacher deep neural network (DNN) models to lightweight target (student) DNN models, enabling the target models to perform better than only training the student model independently. Some previous works focus on using a small, learnable teacher (guide) model to improve the robustness of a student model. Since a learnable guide model starts learning from scratch, maintaining its optimal state for effective knowledge transfer during co-training is challenging. Therefore, we propose a novel Adaptive Guidance Adversarial Training (AdaGAT) method. Our method, AdaGAT, dynamically adjusts the training state of the guide model to install robustness to the target model. Specifically, we develop two separate loss functions as part of the AdaGAT method, allowing the guide model to participate more actively in backpropagation to achieve its optimal state. We evaluated our approach via extensive experiments on three datasets: CIFAR-10, CIFAR-100, and TinyImageNet, using the WideResNet-34-10 model as the target model. Our observations reveal that appropriately adjusting the guide model within a certain accuracy range enhances the target model's robustness across various adversarial attacks compared to a variety of baseline models.