Generalized Majorization-Minimization
This work addresses optimization challenges in machine learning for researchers and practitioners, offering an incremental improvement over existing MM methods.
The paper tackles the problem of non-convex optimization by relaxing the touching constraint in Majorization-Minimization, proposing Generalized Majorization-Minimization (G-MM) as a more flexible framework that can incorporate biases without altering the objective. Empirically, G-MM algorithms consistently outperform MM counterparts in optimizing non-convex objectives and are less sensitive to initialization.
Non-convex optimization is ubiquitous in machine learning. Majorization-Minimization (MM) is a powerful iterative procedure for optimizing non-convex functions that works by optimizing a sequence of bounds on the function. In MM, the bound at each iteration is required to \emph{touch} the objective function at the optimizer of the previous bound. We show that this touching constraint is unnecessary and overly restrictive. We generalize MM by relaxing this constraint, and propose a new optimization framework, named Generalized Majorization-Minimization (G-MM), that is more flexible. For instance, G-MM can incorporate application-specific biases into the optimization procedure without changing the objective function. We derive G-MM algorithms for several latent variable models and show empirically that they consistently outperform their MM counterparts in optimizing non-convex objectives. In particular, G-MM algorithms appear to be less sensitive to initialization.