U-Statistics for Importance-Weighted Variational Inference
This work addresses a specific bottleneck in variational inference for machine learning practitioners, offering an incremental improvement over existing methods.
The paper tackles the problem of high variance in gradient estimation for importance-weighted variational inference by proposing the use of U-statistics to average overlapping batches, leading to modest to significant performance improvements with little computational cost.
We propose the use of U-statistics to reduce variance for gradient estimation in importance-weighted variational inference. The key observation is that, given a base gradient estimator that requires $m > 1$ samples and a total of $n > m$ samples to be used for estimation, lower variance is achieved by averaging the base estimator on overlapping batches of size $m$ than disjoint batches, as currently done. We use classical U-statistic theory to analyze the variance reduction, and propose novel approximations with theoretical guarantees to ensure computational efficiency. We find empirically that U-statistic variance reduction can lead to modest to significant improvements in inference performance on a range of models, with little computational cost.