Natural Evolution Strategies as a Black Box Estimator for Stochastic Variational Inference
This addresses a bottleneck in Bayesian inference for machine learning practitioners by allowing more flexible model designs, though it is an incremental improvement over existing methods.
The paper tackled the limitation of variational autoencoders (VAEs) requiring the reparameterization trick for gradient estimation, which restricts model types, by proposing a natural evolution strategies estimator that removes this assumption, enabling the creation of previously impossible models.
Stochastic variational inference and its derivatives in the form of variational autoencoders enjoy the ability to perform Bayesian inference on large datasets in an efficient manner. However, performing inference with a VAE requires a certain design choice (i.e. reparameterization trick) to allow unbiased and low variance gradient estimation, restricting the types of models that can be created. To overcome this challenge, an alternative estimator based on natural evolution strategies is proposed. This estimator does not make assumptions about the kind of distributions used, allowing for the creation of models that would otherwise not have been possible under the VAE framework.