Scaled Conjugate Gradient Method for Nonconvex Optimization in Deep Neural Networks
This is an incremental improvement for researchers and practitioners in deep learning optimization, offering faster convergence in specific applications.
The authors tackled the problem of nonconvex optimization in deep neural networks by proposing a scaled conjugate gradient method that accelerates existing adaptive methods using stochastic gradients. The result showed it minimizes training loss faster in image/text classification and achieved the lowest Frechet inception distance score in GAN training among adaptive methods.
A scaled conjugate gradient method that accelerates existing adaptive methods utilizing stochastic gradients is proposed for solving nonconvex optimization problems with deep neural networks. It is shown theoretically that, whether with constant or diminishing learning rates, the proposed method can obtain a stationary point of the problem. Additionally, its rate of convergence with diminishing learning rates is verified to be superior to that of the conjugate gradient method. The proposed method is shown to minimize training loss functions faster than the existing adaptive methods in practical applications of image and text classification. Furthermore, in the training of generative adversarial networks, one version of the proposed method achieved the lowest Frechet inception distance score among those of the adaptive methods.