Adaptive Exponential Integration for Stable Gaussian Mixture Black-Box Variational Inference
This work addresses stability issues in variational inference for practitioners in Bayesian statistics and machine learning, though it is incremental as it builds on existing methods with novel adaptations.
The paper tackled the instability and inefficiency of black-box variational inference with Gaussian mixtures by developing a stable framework combining affine-invariant preconditioning, an exponential integrator, and adaptive time stepping, resulting in proven exponential convergence for Gaussian posteriors and demonstrated effectiveness on multimodal distributions and inverse problems.
Black-box variational inference (BBVI) with Gaussian mixture families offers a flexible approach for approximating complex posterior distributions without requiring gradients of the target density. However, standard numerical optimization methods often suffer from instability and inefficiency. We develop a stable and efficient framework that combines three key components: (1) affine-invariant preconditioning via natural gradient formulations, (2) an exponential integrator that unconditionally preserves the positive definiteness of covariance matrices, and (3) adaptive time stepping to ensure stability and to accommodate distinct warm-up and convergence phases. The proposed approach has natural connections to manifold optimization and mirror descent. For Gaussian posteriors, we prove exponential convergence in the noise-free setting and almost-sure convergence under Monte Carlo estimation, rigorously justifying the necessity of adaptive time stepping. Numerical experiments on multimodal distributions, Neal's multiscale funnel, and a PDE-based Bayesian inverse problem for Darcy flow demonstrate the effectiveness of the proposed method.