Particle-based Energetic Variational Inference
This work addresses variational inference for probabilistic modeling, offering a new framework that is incremental as it builds on and generalizes existing particle-based methods.
The authors tackled the problem of variational inference by introducing a new framework called energetic variational inference (EVI), which minimizes the objective based on an energy-dissipation law and can derive existing methods like SVGD while enabling new schemes. They proposed a specific 'Approximation-then-Variation' scheme that maintains variational structure at the particle level and significantly decreases KL-divergence per iteration, with numerical experiments showing it outperforms some existing ParVI methods in fidelity to the target distribution.
We introduce a new variational inference (VI) framework, called energetic variational inference (EVI). It minimizes the VI objective function based on a prescribed energy-dissipation law. Using the EVI framework, we can derive many existing Particle-based Variational Inference (ParVI) methods, including the popular Stein Variational Gradient Descent (SVGD) approach. More importantly, many new ParVI schemes can be created under this framework. For illustration, we propose a new particle-based EVI scheme, which performs the particle-based approximation of the density first and then uses the approximated density in the variational procedure, or "Approximation-then-Variation" for short. Thanks to this order of approximation and variation, the new scheme can maintain the variational structure at the particle level, and can significantly decrease the KL-divergence in each iteration. Numerical experiments show the proposed method outperforms some existing ParVI methods in terms of fidelity to the target distribution.