Towards Better Generalization: BP-SVRG in Training Deep Neural Networks
This work addresses generalization issues in training deep neural networks, but it is incremental as it builds on existing SVRG methods with minor modifications.
The paper tackled the problem of improving generalization in deep neural networks by modifying the Batch-SVRG optimization algorithm, resulting in the proposed BP-SVRG method that shows better generalization performance than B-SVRG and SGD in some scenarios.
Stochastic variance-reduced gradient (SVRG) is a classical optimization method. Although it is theoretically proved to have better convergence performance than stochastic gradient descent (SGD), the generalization performance of SVRG remains open. In this paper we investigate the effects of some training techniques, mini-batching and learning rate decay, on the generalization performance of SVRG, and verify the generalization performance of Batch-SVRG (B-SVRG). In terms of the relationship between optimization and generalization, we believe that the average norm of gradients on each training sample as well as the norm of average gradient indicate how flat the landscape is and how well the model generalizes. Based on empirical observations of such metrics, we perform a sign switch on B-SVRG and derive a practical algorithm, BatchPlus-SVRG (BP-SVRG), which is numerically shown to enjoy better generalization performance than B-SVRG, even SGD in some scenarios of deep neural networks.