Improving Adversarial Robustness of DEQs with Explicit Regulations Along the Neural Dynamics
This work addresses adversarial robustness for DEQ models, which is crucial for their reliable deployment, but it is incremental as it builds on existing adversarial training frameworks.
The paper tackles the problem of adversarial robustness in deep equilibrium (DEQ) models by addressing under-regulation of intermediate states during adversarial training, resulting in substantial robustness improvements that outperform strong deep network baselines.
Deep equilibrium (DEQ) models replace the multiple-layer stacking of conventional deep networks with a fixed-point iteration of a single-layer transformation. Having been demonstrated to be competitive in a variety of real-world scenarios, the adversarial robustness of general DEQs becomes increasingly crucial for their reliable deployment. Existing works improve the robustness of general DEQ models with the widely-used adversarial training (AT) framework, but they fail to exploit the structural uniquenesses of DEQ models. To this end, we interpret DEQs through the lens of neural dynamics and find that AT under-regulates intermediate states. Besides, the intermediate states typically provide predictions with a high prediction entropy. Informed by the correlation between the entropy of dynamical systems and their stability properties, we propose reducing prediction entropy by progressively updating inputs along the neural dynamics. During AT, we also utilize random intermediate states to compute the loss function. Our methods regulate the neural dynamics of DEQ models in this manner. Extensive experiments demonstrate that our methods substantially increase the robustness of DEQ models and even outperform the strong deep network baselines.