Robust Experts: the Effect of Adversarial Training on CNNs with Sparse Mixture-of-Experts Layers
This addresses the problem of adversarial robustness for CNN users, offering an incremental improvement through a hybrid method that increases capacity without extra inference cost.
The paper tackles improving robustness of CNNs against adversarial attacks by using sparse mixture-of-experts layers, finding that inserting a single MoE layer in ResNet on CIFAR-100 leads to consistent robustness gains under PGD and AutoPGD attacks when combined with adversarial training, with some individual experts outperforming the full model due to specialization.
Robustifying convolutional neural networks (CNNs) against adversarial attacks remains challenging and often requires resource-intensive countermeasures. We explore the use of sparse mixture-of-experts (MoE) layers to improve robustness by replacing selected residual blocks or convolutional layers, thereby increasing model capacity without additional inference cost. On ResNet architectures trained on CIFAR-100, we find that inserting a single MoE layer in the deeper stages leads to consistent improvements in robustness under PGD and AutoPGD attacks when combined with adversarial training. Furthermore, we discover that when switch loss is used for balancing, it causes routing to collapse onto a small set of overused experts, thereby concentrating adversarial training on these paths and inadvertently making them more robust. As a result, some individual experts outperform the gated MoE model in robustness, suggesting that robust subpaths emerge through specialization. Our code is available at https://github.com/KASTEL-MobilityLab/robust-sparse-moes.