Learning Mixture Density via Natural Gradient Expectation Maximization
This work addresses optimization challenges in mixture density networks for probabilistic modeling, offering a practical improvement for researchers and practitioners in machine learning.
The paper tackles the slow convergence and mode collapse in training mixture density networks by introducing natural gradient expectation maximization (nGEM), which achieves up to 10x faster convergence with minimal computational overhead and scales effectively to high-dimensional data.
Mixture density networks are neural networks that produce Gaussian mixtures to represent continuous multimodal conditional densities. Standard training procedures involve maximum likelihood estimation using the negative log-likelihood (NLL) objective, which suffers from slow convergence and mode collapse. In this work, we improve the optimization of mixture density networks by integrating their information geometry. Specifically, we interpret mixture density networks as deep latent-variable models and analyze them through an expectation maximization framework, which reveals surprising theoretical connections to natural gradient descent. We then exploit such connections to derive the natural gradient expectation maximization (nGEM) objective. We show that empirically nGEM achieves up to 10$\times$ faster convergence while adding almost zerocomputational overhead, and scales well to high-dimensional data where NLL otherwise fails.