MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters
This addresses the problem of expensive hyperparameter tuning for ML practitioners, though it appears incremental as an adaptive wrapper for existing optimizers.
The paper tackles the challenge of optimizing meta-parameters like learning rates in machine learning by introducing MetaOptimize, a framework that dynamically adjusts step sizes during training to minimize a regret-based objective, achieving performance comparable to best hand-crafted schedules across diverse tasks.
We address the challenge of optimizing meta-parameters (hyperparameters) in machine learning, a key factor for efficient training and high model performance. Rather than relying on expensive meta-parameter search methods, we introduce MetaOptimize: a dynamic approach that adjusts meta-parameters, particularly step sizes (also known as learning rates), during training. More specifically, MetaOptimize can wrap around any first-order optimization algorithm, tuning step sizes on the fly to minimize a specific form of regret that considers the long-term impact of step sizes on training, through a discounted sum of future losses. We also introduce lower-complexity variants of MetaOptimize that, in conjunction with its adaptability to various optimization algorithms, achieve performance comparable to those of the best hand-crafted learning rate schedules across diverse machine learning tasks.