A unified view of likelihood ratio and reparameterization gradients and an optimal importance sampling scheme
This work addresses a foundational issue in machine and reinforcement learning by offering insights into gradient estimation methods, which is incremental as it builds on existing estimators but introduces novel theoretical connections and practical improvements.
The paper tackled the problem of understanding and improving gradient estimators in machine learning by providing a unified theoretical view of likelihood ratio and reparameterization gradients, leading to the derivation of optimal importance sampling schemes that significantly reduce variance, with improvements being modest for Gaussian distributions but arbitrarily large for others like Beta distributions, enabling competitive performance in evolution strategies.
Reparameterization (RP) and likelihood ratio (LR) gradient estimators are used throughout machine and reinforcement learning; however, they are usually explained as simple mathematical tricks without providing any insight into their nature. We use a first principles approach to explain LR and RP, and show a connection between the two via the divergence theorem. The theory motivated us to derive optimal importance sampling schemes to reduce LR gradient variance. Our newly derived distributions have analytic probability densities and can be directly sampled from. The improvement for Gaussian target distributions was modest, but for other distributions such as a Beta distribution, our method could lead to arbitrarily large improvements, and was crucial to obtain competitive performance in evolution strategies experiments.