The Perturbed Prox-Preconditioned SPIDER algorithm for EM-based large scale learning
This work addresses scalability and computational bottlenecks for researchers and practitioners using EM in large-scale machine learning, though it is incremental as it builds on the SPIDER-EM algorithm.
The paper tackles the problem of intractable E-steps and non-smooth regularization in large-scale Expectation Maximization (EM) learning by proposing the Perturbed Prox-Preconditioned SPIDER (3P-SPIDER) algorithm, which outperforms other incremental EM methods in numerical experiments.
Incremental Expectation Maximization (EM) algorithms were introduced to design EM for the large scale learning framework by avoiding the full data set to be processed at each iteration. Nevertheless, these algorithms all assume that the conditional expectations of the sufficient statistics are explicit. In this paper, we propose a novel algorithm named Perturbed Prox-Preconditioned SPIDER (3P-SPIDER), which builds on the Stochastic Path Integral Differential EstimatoR EM (SPIDER-EM) algorithm. The 3P-SPIDER algorithm addresses many intractabilities of the E-step of EM; it also deals with non-smooth regularization and convex constraint set. Numerical experiments show that 3P-SPIDER outperforms other incremental EM methods and discuss the role of some design parameters.