Interpolation between Residual and Non-Residual Networks
This work provides a unified framework for interpreting residual and non-residual networks, with incremental improvements in robustness for image classification tasks.
The authors tackled the unclear relationship between ordinary differential equations (ODEs) and non-residual convolutional neural networks (CNNs) by proposing a damped ODE model that interpolates between ResNet and CNN architectures, resulting in improved accuracy and robustness on image classification benchmarks against noise and adversarial attacks.
Although ordinary differential equations (ODEs) provide insights for designing network architectures, its relationship with the non-residual convolutional neural networks (CNNs) is still unclear. In this paper, we present a novel ODE model by adding a damping term. It can be shown that the proposed model can recover both a ResNet and a CNN by adjusting an interpolation coefficient. Therefore, the damped ODE model provides a unified framework for the interpretation of residual and non-residual networks. The Lyapunov analysis reveals better stability of the proposed model, and thus yields robustness improvement of the learned networks. Experiments on a number of image classification benchmarks show that the proposed model substantially improves the accuracy of ResNet and ResNeXt over the perturbed inputs from both stochastic noise and adversarial attack methods. Moreover, the loss landscape analysis demonstrates the improved robustness of our method along the attack direction.