HesScale: Scalable Computation of Hessian Diagonals
This work addresses a bottleneck in scalable second-order optimization for machine learning practitioners, though it is incremental as it builds on existing Hessian approximation methods.
The paper tackles the problem of expensive Hessian computation in second-order optimization by developing HesScale, a scalable method for approximating Hessian diagonals with computational complexity equal to backpropagation, achieving high approximation accuracy in supervised classification tasks.
Second-order optimization uses curvature information about the objective function, which can help in faster convergence. However, such methods typically require expensive computation of the Hessian matrix, preventing their usage in a scalable way. The absence of efficient ways of computation drove the most widely used methods to focus on first-order approximations that do not capture the curvature information. In this paper, we develop HesScale, a scalable approach to approximating the diagonal of the Hessian matrix, to incorporate second-order information in a computationally efficient manner. We show that HesScale has the same computational complexity as backpropagation. Our results on supervised classification show that HesScale achieves high approximation accuracy, allowing for scalable and efficient second-order optimization.