Towards Constituting Mathematical Structures for Learning to Optimize
This work addresses generalization issues in L2O, a technique for automating optimization algorithms, which is incremental as it builds on existing L2O approaches by incorporating mathematical structure.
The paper tackled the problem of Learning to Optimize (L2O) models overfitting and generalizing poorly to out-of-distribution data by deriving basic mathematical conditions for successful update rules and proposing a novel mathematics-inspired L2O model, which demonstrated superior empirical performance in numerical simulations.
Learning to Optimize (L2O), a technique that utilizes machine learning to learn an optimization algorithm automatically from data, has gained arising attention in recent years. A generic L2O approach parameterizes the iterative update rule and learns the update direction as a black-box network. While the generic approach is widely applicable, the learned model can overfit and may not generalize well to out-of-distribution test sets. In this paper, we derive the basic mathematical conditions that successful update rules commonly satisfy. Consequently, we propose a novel L2O model with a mathematics-inspired structure that is broadly applicable and generalized well to out-of-distribution problems. Numerical simulations validate our theoretical findings and demonstrate the superior empirical performance of the proposed L2O model.