Towards Robust ResNet: A Small Step but A Giant Leap
This work addresses robustness issues in deep learning models like ResNet, but it is incremental as it builds on existing dynamical system interpretations.
The paper tackles the problem of improving the robustness of ResNet by analyzing it through a dynamical system perspective, showing that a small step factor in the Euler method enhances both training and generalization robustness, with empirical validation on CIFAR-10 and AG-NEWS datasets.
This paper presents a simple yet principled approach to boosting the robustness of the residual network (ResNet) that is motivated by the dynamical system perspective. Namely, a deep neural network can be interpreted using a partial differential equation, which naturally inspires us to characterize ResNet by an explicit Euler method. Our analytical studies reveal that the step factor h in the Euler method is able to control the robustness of ResNet in both its training and generalization. Specifically, we prove that a small step factor h can benefit the training robustness for back-propagation; from the view of forward-propagation, a small h can aid in the robustness of the model generalization. A comprehensive empirical evaluation on both vision CIFAR-10 and text AG-NEWS datasets confirms that a small h aids both the training and generalization robustness.