Analytic expressions for the output evolution of a deep neural network
This work provides theoretical insights into generalization in deep learning, but it is incremental as it builds on existing stochastic optimization and regularization concepts without introducing a new paradigm.
The authors tackled the problem of understanding how stochastic training affects deep neural networks by deriving analytic expressions for expected weights and outputs, revealing that noise in early training acts like a linear model improving generalization without explicit regularization, while in later stages it regularizes the output function, as shown through analysis of the weight Hessian.
We present a novel methodology based on a Taylor expansion of the network output for obtaining analytical expressions for the expected value of the network weights and output under stochastic training. Using these analytical expressions the effects of the hyperparameters and the noise variance of the optimization algorithm on the performance of the deep neural network are studied. In the early phases of training with a small noise coefficient, the output is equivalent to a linear model. In this case the network can generalize better due to the noise preventing the output from fully converging on the train data, however the noise does not result in any explicit regularization. In the later training stages, when higher order approximations are required, the impact of the noise becomes more significant, i.e. in a model which is non-linear in the weights noise can regularize the output function resulting in better generalization as witnessed by its influence on the weight Hessian, a commonly used metric for generalization capabilities.