Stochastic Training of Residual Networks: a Differential Equation Viewpoint
This work provides a theoretical framework for stochastic training in deep learning, potentially aiding in designing more reliable and efficient training strategies, though it is incremental in building on existing differential equation viewpoints.
The paper tackles the problem of understanding stochastic training in neural networks by showing that residual networks with noise injection approximate stochastic differential equations, linking training to optimal control of backward Kolmogorov's equations. As an example, they propose using Bernoulli dropout in residual networks and validate it with experiments on an image classification task.
During the last few years, significant attention has been paid to the stochastic training of artificial neural networks, which is known as an effective regularization approach that helps improve the generalization capability of trained models. In this work, the method of modified equations is applied to show that the residual network and its variants with noise injection can be regarded as weak approximations of stochastic differential equations. Such observations enable us to bridge the stochastic training processes with the optimal control of backward Kolmogorov's equations. This not only offers a novel perspective on the effects of regularization from the loss landscape viewpoint but also sheds light on the design of more reliable and efficient stochastic training strategies. As an example, we propose a new way to utilize Bernoulli dropout within the plain residual network architecture and conduct experiments on a real-world image classification task to substantiate our theoretical findings.