Don't Be So Positive: Negative Step Sizes in Second-Order Methods
This addresses a bottleneck in optimization for machine learning practitioners by improving the performance of second-order methods in neural network training, though it is incremental as it builds on existing methods.
The paper tackles the inefficiency of second-order methods in neural network optimization by proposing the use of negative step sizes to leverage negative curvature information, which is often discarded, and shows that this approach is globally convergent and more effective than common Hessian modification methods in experiments.
The value of second-order methods lies in the use of curvature information. Yet, this information is costly to extract and once obtained, valuable negative curvature information is often discarded so that the method is globally convergent. This limits the effectiveness of second-order methods in modern machine learning. In this paper, we show that second-order and second-order-like methods are promising optimizers for neural networks provided that we add one ingredient: negative step sizes. We show that under very general conditions, methods that produce ascent directions are globally convergent when combined with a Wolfe line search that allows both positive and negative step sizes. We experimentally demonstrate that using negative step sizes is often more effective than common Hessian modification methods.