Newton Methods for Convolutional Neural Networks
This work addresses the optimization problem for deep learning practitioners by enabling Newton methods in CNNs, though it is incremental as it adapts existing techniques to a new network type.
The authors tackled the challenge of applying Newton methods to Convolutional Neural Networks (CNNs), which had not been thoroughly investigated due to operational complexities, by detailing building blocks like gradient and Jacobian evaluation, and demonstrated that their efficient MATLAB implementation achieves competitive test accuracy.
Deep learning involves a difficult non-convex optimization problem, which is often solved by stochastic gradient (SG) methods. While SG is usually effective, it may not be robust in some situations. Recently, Newton methods have been investigated as an alternative optimization technique, but nearly all existing studies consider only fully-connected feedforward neural networks. They do not investigate other types of networks such as Convolutional Neural Networks (CNN), which are more commonly used in deep-learning applications. One reason is that Newton methods for CNN involve complicated operations, and so far no works have conducted a thorough investigation. In this work, we give details of all building blocks including function, gradient, and Jacobian evaluation, and Gauss-Newton matrix-vector products. These basic components are very important because with them further developments of Newton methods for CNN become possible. We show that an efficient MATLAB implementation can be done in just several hundred lines of code and demonstrate that the Newton method gives competitive test accuracy.