Differentiable Antithetic Sampling for Variance Reduction in Stochastic Variational Inference
This addresses the computational efficiency challenge in variational inference for machine learning practitioners, though it appears incremental as it builds on existing antithetic sampling concepts.
The paper tackles the problem of high variance in stochastic variational inference by introducing a differentiable antithetic sampling method that generates correlated samples matching the true moments of the importance distribution, resulting in improved performance for learning deep generative models.
Stochastic optimization techniques are standard in variational inference algorithms. These methods estimate gradients by approximating expectations with independent Monte Carlo samples. In this paper, we explore a technique that uses correlated, but more representative , samples to reduce estimator variance. Specifically, we show how to generate antithetic samples that match sample moments with the true moments of an underlying importance distribution. Combining a differentiable antithetic sampler with modern stochastic variational inference, we showcase the effectiveness of this approach for learning a deep generative model.