Make Optimization Once and for All with Fine-grained Guidance
This work addresses scalability and generalization issues in optimization for machine learning practitioners, though it appears incremental as it builds on existing L2O paradigms with a novel framework.
The paper tackles the problem of limited scalability and generalization in Learning to Optimize (L2O) methods by proposing Diff-L2O, a general framework that augments sampled solutions from a broader perspective rather than relying on local updates. The result shows strong compatibility with minute-level training compared to hour-level baselines, and a generalization bound demonstrates that sample diversity improves performance.
Learning to Optimize (L2O) enhances optimization efficiency with integrated neural networks. L2O paradigms achieve great outcomes, e.g., refitting optimizer, generating unseen solutions iteratively or directly. However, conventional L2O methods require intricate design and rely on specific optimization processes, limiting scalability and generalization. Our analyses explore general framework for learning optimization, called Diff-L2O, focusing on augmenting sampled solutions from a wider view rather than local updates in real optimization process only. Meanwhile, we give the related generalization bound, showing that the sample diversity of Diff-L2O brings better performance. This bound can be simply applied to other fields, discussing diversity, mean-variance, and different tasks. Diff-L2O's strong compatibility is empirically verified with only minute-level training, comparing with other hour-levels.