A Dynamical Systems Approach for Convergence of the Bayesian EM Algorithm
This work addresses the gap in systems and control-based analysis for machine learning algorithms, offering new convergence guarantees for MAP-EM, which is incremental as it expands existing conditions rather than introducing a new method.
The paper tackles the analysis of optimization algorithms in machine learning by applying discrete-time Lyapunov stability theory to study the convergence of the maximum a posteriori expectation-maximization (MAP-EM) algorithm, showing conditions for convergence and potential fast linear or quadratic convergence under additional assumptions.
Out of the recent advances in systems and control (S\&C)-based analysis of optimization algorithms, not enough work has been specifically dedicated to machine learning (ML) algorithms and its applications. This paper addresses this gap by illustrating how (discrete-time) Lyapunov stability theory can serve as a powerful tool to aid, or even lead, in the analysis (and potential design) of optimization algorithms that are not necessarily gradient-based. The particular ML problem that this paper focuses on is that of parameter estimation in an incomplete-data Bayesian framework via the popular optimization algorithm known as maximum a posteriori expectation-maximization (MAP-EM). Following first principles from dynamical systems stability theory, conditions for convergence of MAP-EM are developed. Furthermore, if additional assumptions are met, we show that fast convergence (linear or quadratic) is achieved, which could have been difficult to unveil without our adopted S\&C approach. The convergence guarantees in this paper effectively expand the set of sufficient conditions for EM applications, thereby demonstrating the potential of similar S\&C-based convergence analysis of other ML algorithms.