POP: Prior-fitted Optimizer Policies
This addresses the problem of hyperparameter tuning in optimization for machine learning practitioners, offering a generalizable solution that is incremental over prior meta-learning approaches.
The paper tackles the sensitivity of gradient-based optimizers to hyperparameters in non-convex settings by introducing POP, a meta-learned optimizer that predicts step sizes based on optimization trajectory context, and it outperforms various methods on a benchmark of 47 functions under matched budget constraints.
Optimization refers to the task of finding extrema of an objective function. Classical gradient-based optimizers are highly sensitive to hyperparameter choices. In highly non-convex settings their performance relies on carefully tuned learning rates, momentum, and gradient accumulation. To address these limitations, we introduce POP (Prior-fitted Optimizer Policies), a meta-learned optimizer that predicts coordinate-wise step sizes conditioned on the contextual information provided in the optimization trajectory. Our model is learned on millions of synthetic optimization problems sampled from a novel prior spanning both convex and non-convex objectives. We evaluate POP on an established benchmark including 47 optimization functions of various complexity, where it consistently outperforms first-order gradient-based methods, non-convex optimization approaches (e.g., evolutionary strategies), Bayesian optimization, and a recent meta-learned competitor under matched budget constraints. Our evaluation demonstrates strong generalization capabilities without task-specific tuning.