Note on the bias and variance of variational inference
This work addresses theoretical understanding of bias in variational inference for machine learning practitioners, but it appears incremental as it builds on existing concepts without introducing a new paradigm.
The paper investigates the connection between the variational gap and the variance of the log-likelihood ratio in variational inference, showing that the bias can be bounded by a dispersion measure and potentially reduced through techniques like averaging and variance reduction.
In this note, we study the relationship between the variational gap and the variance of the (log) likelihood ratio. We show that the gap can be upper bounded by some form of dispersion measure of the likelihood ratio, which suggests the bias of variational inference can be reduced by making the distribution of the likelihood ratio more concentrated, such as via averaging and variance reduction.