Improving Explorability in Variational Inference with Annealed Variational Objectives
This addresses optimization limitations in variational inference for probabilistic modeling researchers, presenting an incremental improvement to hierarchical variational methods.
The paper tackles the problem of variational inference optimization limiting learned posterior distributions, showing drawbacks of unimodal bias and introducing Annealed Variational Objectives (AVO) to incorporate energy tempering into training. The method demonstrates robustness to deterministic warm-up and benefits of latent space exploration in experiments.
Despite the advances in the representational capacity of approximate distributions for variational inference, the optimization process can still limit the density that is ultimately learned. We demonstrate the drawbacks of biasing the true posterior to be unimodal, and introduce Annealed Variational Objectives (AVO) into the training of hierarchical variational methods. Inspired by Annealed Importance Sampling, the proposed method facilitates learning by incorporating energy tempering into the optimization objective. In our experiments, we demonstrate our method's robustness to deterministic warm up, and the benefits of encouraging exploration in the latent space.