Optimizing Approximate Leave-one-out Cross-validation to Tune Hyperparameters
This work provides a more efficient method for hyperparameter tuning using approximate leave-one-out cross-validation for practitioners working with regularized models, potentially speeding up model development.
This paper addresses the problem of optimizing approximate leave-one-out (ALO) cross-validation for hyperparameter tuning in regularized models. The authors derive efficient formulas for computing the gradient and Hessian of ALO, enabling the use of second-order optimizers to find optimal hyperparameters for models like regularized logistic regression and ridge regression on real-world datasets.
For a large class of regularized models, leave-one-out cross-validation can be efficiently estimated with an approximate leave-one-out formula (ALO). We consider the problem of adjusting hyperparameters so as to optimize ALO. We derive efficient formulas to compute the gradient and hessian of ALO and show how to apply a second-order optimizer to find hyperparameters. We demonstrate the usefulness of the proposed approach by finding hyperparameters for regularized logistic regression and ridge regression on various real-world data sets.