A predictor-corrector method for the training of deep neural networks
This work addresses the problem of expensive training times for deep learning practitioners, but it is incremental as it builds on existing SGD methods with a modest speedup.
The paper tackles the high computational cost of training deep neural networks by introducing a predictor-corrector method that alternates predictor and corrector passes using standard stochastic gradient descent with backpropagation, achieving a 9% time improvement on the CIFAR-10 dataset without loss in validation accuracy.
The training of deep neural nets is expensive. We present a predictor- corrector method for the training of deep neural nets. It alternates a predictor pass with a corrector pass using stochastic gradient descent with backpropagation such that there is no loss in validation accuracy. No special modifications to SGD with backpropagation is required by this methodology. Our experiments showed a time improvement of 9% on the CIFAR-10 dataset.