Empirical Evaluation of Biased Methods for Alpha Divergence Minimization
This work addresses the practical limitations of biased alpha-divergence minimization methods for researchers and practitioners in machine learning, highlighting incremental insights into computational inefficiencies.
The paper empirically evaluates biased methods for minimizing alpha-divergence, finding that these methods strongly bias solutions towards minimizing the exclusive KL-divergence and require impractically large computation in high dimensions to mitigate this bias and achieve true alpha-divergence minimization.
In this paper we empirically evaluate biased methods for alpha-divergence minimization. In particular, we focus on how the bias affects the final solutions found, and how this depends on the dimensionality of the problem. We find that (i) solutions returned by these methods appear to be strongly biased towards minimizers of the traditional "exclusive" KL-divergence, KL(q||p), and (ii) in high dimensions, an impractically large amount of computation is needed to mitigate this bias and obtain solutions that actually minimize the alpha-divergence of interest.