DARD: Dice Adversarial Robustness Distillation against Adversarial Attacks
This addresses security challenges in deep learning by improving adversarial robustness for real-world applications, though it is incremental as it builds on existing knowledge distillation and adversarial training methods.
The paper tackles the trade-off between robustness and accuracy in adversarial defense by distilling robustness from large teacher models into compact student models, achieving superior robustness and standard accuracy compared to adversarially trained networks.
Deep learning models are vulnerable to adversarial examples, posing critical security challenges in real-world applications. While Adversarial Training (AT ) is a widely adopted defense mechanism to enhance robustness, it often incurs a trade-off by degrading performance on unperturbed, natural data. Recent efforts have highlighted that larger models exhibit enhanced robustness over their smaller counterparts. In this paper, we empirically demonstrate that such robustness can be systematically distilled from large teacher models into compact student models. To achieve better performance, we introduce Dice Adversarial Robustness Distillation (DARD), a novel method designed to transfer robustness through a tailored knowledge distillation paradigm. Additionally, we propose Dice Projected Gradient Descent (DPGD), an adversarial example generalization method optimized for effective attack. Our extensive experiments demonstrate that the DARD approach consistently outperforms adversarially trained networks with the same architecture, achieving superior robustness and standard accuracy.