Adversarial Training via Adaptive Knowledge Amalgamation of an Ensemble of Teachers
This addresses robustness challenges in deep neural networks for security applications, but it is incremental as it builds on existing adversarial training and distillation techniques.
The paper tackles the problem of adversarial training's limitations in achieving robustness for smaller models and poor generalization to unforeseen attacks by introducing AT-AKA, which adaptively amalgamates knowledge from an ensemble of teachers to train a robust student, showing superior efficacy over existing methods against attacks like AutoAttack.
Adversarial training (AT) is a popular method for training robust deep neural networks (DNNs) against adversarial attacks. Yet, AT suffers from two shortcomings: (i) the robustness of DNNs trained by AT is highly intertwined with the size of the DNNs, posing challenges in achieving robustness in smaller models; and (ii) the adversarial samples employed during the AT process exhibit poor generalization, leaving DNNs vulnerable to unforeseen attack types. To address these dual challenges, this paper introduces adversarial training via adaptive knowledge amalgamation of an ensemble of teachers (AT-AKA). In particular, we generate a diverse set of adversarial samples as the inputs to an ensemble of teachers; and then, we adaptively amalgamate the logtis of these teachers to train a generalized-robust student. Through comprehensive experiments, we illustrate the superior efficacy of AT-AKA over existing AT methods and adversarial robustness distillation techniques against cutting-edge attacks, including AutoAttack.