Sample-wise Adaptive Weighting for Transfer Consistency in Adversarial Distillation
This work addresses the challenge of efficiently transferring adversarial robustness from large teachers to compact students in machine learning, offering a novel method to overcome robust saturation, which is incremental but impactful for domain-specific applications like adversarial training.
The paper tackles the problem of robust saturation in adversarial distillation, where stronger teachers do not necessarily yield more robust students, by identifying adversarial transferability as a key factor and proposing Sample-wise Adaptive Adversarial Distillation (SAAD) to reweight training examples, resulting in consistent improvements in AutoAttack robustness across CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets.
Adversarial distillation in the standard min-max adversarial training framework aims to transfer adversarial robustness from a large, robust teacher network to a compact student. However, existing work often neglects to incorporate state-of-the-art robust teachers. Through extensive analysis, we find that stronger teachers do not necessarily yield more robust students-a phenomenon known as robust saturation. While typically attributed to capacity gaps, we show that such explanations are incomplete. Instead, we identify adversarial transferability-the fraction of student-crafted adversarial examples that remain effective against the teacher-as a key factor in successful robustness transfer. Based on this insight, we propose Sample-wise Adaptive Adversarial Distillation (SAAD), which reweights training examples by their measured transferability without incurring additional computational cost. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet show that SAAD consistently improves AutoAttack robustness over prior methods. Our code is available at https://github.com/HongsinLee/saad.