SketchySGD: Reliable Stochastic Optimization via Randomized Curvature Estimates
This addresses the need for robust and efficient optimization in machine learning, particularly for large-scale or ill-conditioned datasets, though it is incremental as it builds on existing stochastic gradient methods.
The paper tackles the problem of unreliable stochastic gradient methods in machine learning by introducing SketchySGD, which uses randomized low-rank Hessian approximations and an automated stepsize to achieve linear convergence and outperform SGD on ill-conditioned problems, as shown in experiments where it solved a logistic regression problem with over 840GB RAM data while competitors failed.
SketchySGD improves upon existing stochastic gradient methods in machine learning by using randomized low-rank approximations to the subsampled Hessian and by introducing an automated stepsize that works well across a wide range of convex machine learning problems. We show theoretically that SketchySGD with a fixed stepsize converges linearly to a small ball around the optimum. Further, in the ill-conditioned setting we show SketchySGD converges at a faster rate than SGD for least-squares problems. We validate this improvement empirically with ridge regression experiments on real data. Numerical experiments on both ridge and logistic regression problems with dense and sparse data, show that SketchySGD equipped with its default hyperparameters can achieve comparable or better results than popular stochastic gradient methods, even when they have been tuned to yield their best performance. In particular, SketchySGD is able to solve an ill-conditioned logistic regression problem with a data matrix that takes more than $840$GB RAM to store, while its competitors, even when tuned, are unable to make any progress. SketchySGD's ability to work out-of-the box with its default hyperparameters and excel on ill-conditioned problems is an advantage over other stochastic gradient methods, most of which require careful hyperparameter tuning (especially of the learning rate) to obtain good performance and degrade in the presence of ill-conditioning.