Learning to Initialize Gradient Descent Using Gradient Descent
This work addresses the problem of finding good initializations for gradient descent, which is crucial for the success and efficiency of optimization, for practitioners working with non-convex problems.
This paper proposes a method to learn initialization rules for gradient descent in non-convex optimization problems from previous solutions. The approach consistently outperforms other initialization techniques across tasks like adversarial example generation, post hoc explanations, and communication spectrum allocation.
Non-convex optimization problems are challenging to solve; the success and computational expense of a gradient descent algorithm or variant depend heavily on the initialization strategy. Often, either random initialization is used or initialization rules are carefully designed by exploiting the nature of the problem class. As a simple alternative to hand-crafted initialization rules, we propose an approach for learning "good" initialization rules from previous solutions. We provide theoretical guarantees that establish conditions that are sufficient in all cases and also necessary in some under which our approach performs better than random initialization. We apply our methodology to various non-convex problems such as generating adversarial examples, generating post hoc explanations for black-box machine learning models, and allocating communication spectrum, and show consistent gains over other initialization techniques.