Towards Understanding the Importance of Noise in Training Neural Networks
This addresses a fundamental theoretical gap in machine learning for researchers, though it is incremental as it focuses on a simplified model.
The paper tackles the problem of understanding why noise helps in training neural networks by proving that perturbed gradient descent with noise annealing can converge to a global optimum in polynomial time for a simple two-layer CNN, escaping spurious local optima, as supported by numerical experiments.
Numerous empirical evidence has corroborated that the noise plays a crucial rule in effective and efficient training of neural networks. The theory behind, however, is still largely unknown. This paper studies this fundamental problem through training a simple two-layer convolutional neural network model. Although training such a network requires solving a nonconvex optimization problem with a spurious local optimum and a global optimum, we prove that perturbed gradient descent and perturbed mini-batch stochastic gradient algorithms in conjunction with noise annealing is guaranteed to converge to a global optimum in polynomial time with arbitrary initialization. This implies that the noise enables the algorithm to efficiently escape from the spurious local optimum. Numerical experiments are provided to support our theory.