Building Robust Ensembles via Margin Boosting
This work addresses adversarial robustness for neural networks, which is critical for security in AI applications, but it appears incremental as it builds on existing ensembling and adversarial training methods.
The paper tackles the problem of adversarial robustness by proposing a principled approach to build robust ensembles via margin boosting, resulting in an algorithm that outperforms existing ensembling techniques and large end-to-end trained models on benchmark datasets, with a byproduct MCE loss that improves state-of-the-art adversarial training when replacing standard cross-entropy loss.
In the context of adversarial robustness, a single model does not usually have enough power to defend against all possible adversarial attacks, and as a result, has sub-optimal robustness. Consequently, an emerging line of work has focused on learning an ensemble of neural networks to defend against adversarial attacks. In this work, we take a principled approach towards building robust ensembles. We view this problem from the perspective of margin-boosting and develop an algorithm for learning an ensemble with maximum margin. Through extensive empirical evaluation on benchmark datasets, we show that our algorithm not only outperforms existing ensembling techniques, but also large models trained in an end-to-end fashion. An important byproduct of our work is a margin-maximizing cross-entropy (MCE) loss, which is a better alternative to the standard cross-entropy (CE) loss. Empirically, we show that replacing the CE loss in state-of-the-art adversarial training techniques with our MCE loss leads to significant performance improvement.