How to Fill the Optimum Set? Population Gradient Descent with Harmless Diversity
This addresses the need for multiple optima in deep learning problems, offering a practical solution for applications requiring diverse outputs, though it is incremental as it builds on existing optimization frameworks.
The paper tackles the problem of finding diverse optimal solutions in machine learning by framing it as a bi-level optimization to maximize diversity within the optimum set, and demonstrates that their method efficiently generates diverse solutions across applications like text-to-image generation and molecular conformation.
Although traditional optimization methods focus on finding a single optimal solution, most objective functions in modern machine learning problems, especially those in deep learning, often have multiple or infinite numbers of optima. Therefore, it is useful to consider the problem of finding a set of diverse points in the optimum set of an objective function. In this work, we frame this problem as a bi-level optimization problem of maximizing a diversity score inside the optimum set of the main loss function, and solve it with a simple population gradient descent framework that iteratively updates the points to maximize the diversity score in a fashion that does not hurt the optimization of the main loss. We demonstrate that our method can efficiently generate diverse solutions on a variety of applications, including text-to-image generation, text-to-mesh generation, molecular conformation generation and ensemble neural network training.