Convergence Analysis for Training Stochastic Neural Networks via Stochastic Gradient Descent
This provides theoretical guarantees for efficient training of SNNs, which is incremental for researchers in stochastic optimization and neural network theory.
The paper tackles the problem of training stochastic neural networks (SNNs) by proving convergence for a novel sample-wise back-propagation method, showing that training steps should scale with the square of the number of layers in convex cases.
In this paper, we carry out numerical analysis to prove convergence of a novel sample-wise back-propagation method for training a class of stochastic neural networks (SNNs). The structure of the SNN is formulated as discretization of a stochastic differential equation (SDE). A stochastic optimal control framework is introduced to model the training procedure, and a sample-wise approximation scheme for the adjoint backward SDE is applied to improve the efficiency of the stochastic optimal control solver, which is equivalent to the back-propagation for training the SNN. The convergence analysis is derived with and without convexity assumption for optimization of the SNN parameters. Especially, our analysis indicates that the number of SNN training steps should be proportional to the square of the number of layers in the convex optimization case. Numerical experiments are carried out to validate the analysis results, and the performance of the sample-wise back-propagation method for training SNNs is examined by benchmark machine learning examples.