Optimizing Variational Representations of Divergences and Accelerating their Statistical Estimation
This work addresses a bottleneck in machine learning for researchers and practitioners using variational methods, offering incremental improvements to accelerate statistical estimation.
The paper tackled the problem of improving the tightness of variational representations of divergences, which are used for training probabilistic models and differentiating data distributions, by developing a methodology that constructs tighter representations through an auxiliary optimization problem, resulting in significantly faster learning and more accurate estimation, often accelerated by nearly an order of magnitude in datasets over 1000 dimensions.
Variational representations of divergences and distances between high-dimensional probability distributions offer significant theoretical insights and practical advantages in numerous research areas. Recently, they have gained popularity in machine learning as a tractable and scalable approach for training probabilistic models and for statistically differentiating between data distributions. Their advantages include: 1) They can be estimated from data as statistical averages. 2) Such representations can leverage the ability of neural networks to efficiently approximate optimal solutions in function spaces. However, a systematic and practical approach to improving the tightness of such variational formulas, and accordingly accelerate statistical learning and estimation from data, is currently lacking. Here we develop such a methodology for building new, tighter variational representations of divergences. Our approach relies on improved objective functionals constructed via an auxiliary optimization problem. Furthermore, the calculation of the functional Hessian of objective functionals unveils the local curvature differences around the common optimal variational solution; this quantifies and orders the tightness gains between different variational representations. Finally, numerical simulations utilizing neural network optimization demonstrate that tighter representations can result in significantly faster learning and more accurate estimation of divergences in both synthetic and real datasets (of more than 1000 dimensions), often accelerated by nearly an order of magnitude.