Newton methods based convolution neural networks using parallel processing
This work addresses training speed issues for deep learning practitioners, but it is incremental as it builds on existing Newton methods with parallelization.
The paper tackles the inefficiency of training convolutional neural networks with uncertain learning rates by using full-data Newton methods instead of sub-sampled ones and implementing parallel processing for mini-batch computations, resulting in improved training time compared to previous approaches.
Training of convolutional neural networks is a high dimensional and a non-convex optimization problem. At present, it is inefficient in situations where parametric learning rates can not be confidently set. Some past works have introduced Newton methods for training deep neural networks. Newton methods for convolutional neural networks involve complicated operations. Finding the Hessian matrix in second-order methods becomes very complex as we mainly use the finite differences method with the image data. Newton methods for convolutional neural networks deals with this by using the sub-sampled Hessian Newton methods. In this paper, we have used the complete data instead of the sub-sampled methods that only handle partial data at a time. Further, we have used parallel processing instead of serial processing in mini-batch computations. The results obtained using parallel processing in this study, outperform the time taken by the previous approach.