Estimating the Hessian by Back-propagating Curvature
This addresses the bottleneck of Hessian computation in machine learning, particularly for applications like neural networks and Score Matching, offering a scalable alternative to inefficient exact methods.
The paper tackles the problem of efficiently computing unbiased approximations of the Hessian for functions in computational graphs, introducing Curvature Propagation (CP) that provides accurate estimates, such as for neural networks, with relatively low computational cost.
In this work we develop Curvature Propagation (CP), a general technique for efficiently computing unbiased approximations of the Hessian of any function that is computed using a computational graph. At the cost of roughly two gradient evaluations, CP can give a rank-1 approximation of the whole Hessian, and can be repeatedly applied to give increasingly precise unbiased estimates of any or all of the entries of the Hessian. Of particular interest is the diagonal of the Hessian, for which no general approach is known to exist that is both efficient and accurate. We show in experiments that CP turns out to work well in practice, giving very accurate estimates of the Hessian of neural networks, for example, with a relatively small amount of work. We also apply CP to Score Matching, where a diagonal of a Hessian plays an integral role in the Score Matching objective, and where it is usually computed exactly using inefficient algorithms which do not scale to larger and more complex models.