On the Behavior of the Expectation-Maximization Algorithm for Mixture Models
This work addresses computational challenges in mixture model inference, extending existing Gaussian results to Laplacian distributions, but is incremental as it builds on prior EM analysis.
The paper tackled the non-convex optimization problem in mixture models by analyzing the Expectation-Maximization (EM) algorithm for Laplacian distributions, proving global optimality and convergence to ground truth parameters, and proposing a novel stochastic method that outperforms Naive EM in numerical experiments.
Finite mixture models are among the most popular statistical models used in different data science disciplines. Despite their broad applicability, inference under these models typically leads to computationally challenging non-convex problems. While the Expectation-Maximization (EM) algorithm is the most popular approach for solving these non-convex problems, the behavior of this algorithm is not well understood. In this work, we focus on the case of mixture of Laplacian (or Gaussian) distribution. We start by analyzing a simple equally weighted mixture of two single dimensional Laplacian distributions and show that every local optimum of the population maximum likelihood estimation problem is globally optimal. Then, we prove that the EM algorithm converges to the ground truth parameters almost surely with random initialization. Our result extends the existing results for Gaussian distribution to Laplacian distribution. Then we numerically study the behavior of mixture models with more than two components. Motivated by our extensive numerical experiments, we propose a novel stochastic method for estimating the mean of components of a mixture model. Our numerical experiments show that our algorithm outperforms the Naive EM algorithm in almost all scenarios.