Universal Majorization-Minimization Algorithms
This provides a general-purpose optimization tool for researchers and practitioners in machine learning and related fields, though it appears incremental as it builds on existing MM and automatic differentiation techniques.
The paper tackles the limitation of traditional majorization-minimization (MM) methods, which require hand-derived majorizers and are only applicable to a few problems, by introducing universal MM optimizers that automatically derive majorizers using Taylor mode automatic differentiation, enabling application to arbitrary problems with convergence from any starting point and no hyperparameter tuning.
Majorization-minimization (MM) is a family of optimization methods that iteratively reduce a loss by minimizing a locally-tight upper bound, called a majorizer. Traditionally, majorizers were derived by hand, and MM was only applicable to a small number of well-studied problems. We present optimizers that instead derive majorizers automatically, using a recent generalization of Taylor mode automatic differentiation. These universal MM optimizers can be applied to arbitrary problems and converge from any starting point, with no hyperparameter tuning.