Proximity Variational Inference
This addresses a key bottleneck in variational inference for researchers and practitioners, offering a flexible solution to enhance optimization robustness, though it is incremental as it builds on existing variational methods.
The paper tackles the sensitivity of variational inference to initialization and local optima by introducing proximity variational inference (PVI), a method that constrains variational parameters to improve optimization, resulting in consistently better local optima and predictive performance in tests on models like Bernoulli factor models and variational autoencoders.
Variational inference is a powerful approach for approximate posterior inference. However, it is sensitive to initialization and can be subject to poor local optima. In this paper, we develop proximity variational inference (PVI). PVI is a new method for optimizing the variational objective that constrains subsequent iterates of the variational parameters to robustify the optimization path. Consequently, PVI is less sensitive to initialization and optimization quirks and finds better local optima. We demonstrate our method on three proximity statistics. We study PVI on a Bernoulli factor model and sigmoid belief network with both real and synthetic data and compare to deterministic annealing (Katahira et al., 2008). We highlight the flexibility of PVI by designing a proximity statistic for Bayesian deep learning models such as the variational autoencoder (Kingma and Welling, 2014; Rezende et al., 2014). Empirically, we show that PVI consistently finds better local optima and gives better predictive performance.