A provably convergent alternating minimization method for mean field inference
This provides a theoretical foundation for a widely used method in Bayesian inference, addressing a known bottleneck in variational approximation.
The paper tackles the lack of convergence guarantees for alternating coordinate minimization in mean-field variational inference by adding a penalization term, resulting in provable convergence to a critical point with closed-form updates and a derived convergence rate.
Mean-Field is an efficient way to approximate a posterior distribution in complex graphical models and constitutes the most popular class of Bayesian variational approximation methods. In most applications, the mean field distribution parameters are computed using an alternate coordinate minimization. However, the convergence properties of this algorithm remain unclear. In this paper, we show how, by adding an appropriate penalization term, we can guarantee convergence to a critical point, while keeping a closed form update at each step. A convergence rate estimate can also be derived based on recent results in non-convex optimization.