Nonlinear Least Squares for Large-Scale Machine Learning using Stochastic Jacobian Estimates
This addresses computational bottlenecks in large-scale machine learning optimization, though it appears incremental as it builds on existing low-rank Hessian properties.
The paper tackles the problem of efficiently solving large-scale nonlinear least squares problems in machine learning by exploiting the low-rank structure of the Hessian when model parameters exceed batch data size, resulting in two algorithms that perform competitively with state-of-the-art methods.
For large nonlinear least squares loss functions in machine learning we exploit the property that the number of model parameters typically exceeds the data in one batch. This implies a low-rank structure in the Hessian of the loss, which enables effective means to compute search directions. Using this property, we develop two algorithms that estimate Jacobian matrices and perform well when compared to state-of-the-art methods.