Ensemble-in-One: Learning Ensemble within Random Gated Networks for Enhanced Adversarial Robustness
This addresses the scalability and deployment challenges of ensemble methods for adversarial defense in real-world applications, though it is incremental as it builds on existing ensemble and adversarial training techniques.
The paper tackles the problem of adversarial attacks on deep learning systems by proposing ensemble-in-one (EIO), a method that trains an ensemble within a random gated network to enhance robustness while maintaining accuracy on clean data, achieving better performance with less computational overhead compared to previous ensemble methods.
Adversarial attacks have rendered high security risks on modern deep learning systems. Adversarial training can significantly enhance the robustness of neural network models by suppressing the non-robust features. However, the models often suffer from significant accuracy loss on clean data. Ensemble training methods have emerged as promising solutions for defending against adversarial attacks by diversifying the vulnerabilities among the sub-models, simultaneously maintaining comparable accuracy as standard training. However, existing ensemble methods are with poor scalability, owing to the rapid complexity increase when including more sub-models in the ensemble. Moreover, in real-world applications, it is difficult to deploy an ensemble with multiple sub-models, owing to the tight hardware resource budget and latency requirement. In this work, we propose ensemble-in-one (EIO), a simple but efficient way to train an ensemble within one random gated network (RGN). EIO augments the original model by replacing the parameterized layers with multi-path random gated blocks (RGBs) to construct a RGN. By diversifying the vulnerability of the numerous paths within the RGN, better robustness can be achieved. It provides high scalability because the paths within an EIO network exponentially increase with the network depth. Our experiments demonstrate that EIO consistently outperforms previous ensemble training methods with even less computational overhead.