Online Hyperparameter Search Interleaved with Proximal Parameter Updates
This addresses the need for efficient hyperparameter optimization in statistical learning, particularly for non-smooth problems, though it appears incremental as it builds on existing proximal gradient methods.
The paper tackles the problem of inefficient hyperparameter tuning for non-smooth cost functions like Lasso regression by developing a method based on proximal gradient structures, resulting in convergence to a local optimum of the Leave-one-out validation error curve with efficient approximations.
There is a clear need for efficient algorithms to tune hyperparameters for statistical learning schemes, since the commonly applied search methods (such as grid search with N-fold cross-validation) are inefficient and/or approximate. Previously existing algorithms that efficiently search for hyperparameters relying on the smoothness of the cost function cannot be applied in problems such as Lasso regression. In this contribution, we develop a hyperparameter optimization method that relies on the structure of proximal gradient methods and does not require a smooth cost function. Such a method is applied to Leave-one-out (LOO)-validated Lasso and Group Lasso to yield efficient, data-driven, hyperparameter optimization algorithms. Numerical experiments corroborate the convergence of the proposed method to a local optimum of the LOO validation error curve, and the efficiency of its approximations.