Stabilizing Bi-Level Hyperparameter Optimization using Moreau-Yosida Regularization
This work addresses stability issues in hyperparameter optimization for machine learning practitioners, representing an incremental improvement over existing methods.
The paper tackled the problem of unstable convergence in bi-level hyperparameter optimization by proposing the Moreau-Yosida regularized Hyperparameter Optimization (MY-HPO) algorithm, resulting in significant improvement in loss values for a fixed computation budget compared to state-of-the-art solvers.
This research proposes to use the Moreau-Yosida envelope to stabilize the convergence behavior of bi-level Hyperparameter optimization solvers, and introduces the new algorithm called Moreau-Yosida regularized Hyperparameter Optimization (MY-HPO) algorithm. Theoretical analysis on the correctness of the MY-HPO solution and initial convergence analysis is also provided. Our empirical results show significant improvement in loss values for a fixed computation budget, compared to the state-of-art bi-level HPO solvers.