Fast approximations of the Jeffreys divergence between univariate Gaussian mixture models via exponential polynomial densities
This work addresses a computational bottleneck for researchers and practitioners in information sciences who need efficient divergence calculations, though it is incremental as it builds on existing approximation techniques.
The paper tackles the problem of approximating the Jeffreys divergence between univariate Gaussian mixture models, which lacks a closed-form solution, by proposing a fast heuristic using exponential polynomial densities. The result shows an improvement in computational time by several orders of magnitude compared to Monte Carlo methods, with reasonable accuracy, especially for mixtures with few modes.
The Jeffreys divergence is a renown symmetrization of the oriented Kullback-Leibler divergence broadly used in information sciences. Since the Jeffreys divergence between Gaussian mixture models is not available in closed-form, various techniques with pros and cons have been proposed in the literature to either estimate, approximate, or lower and upper bound this divergence. In this paper, we propose a simple yet fast heuristic to approximate the Jeffreys divergence between two univariate Gaussian mixtures with arbitrary number of components. Our heuristic relies on converting the mixtures into pairs of dually parameterized probability densities belonging to an exponential family. In particular, we consider the versatile polynomial exponential family densities, and design a divergence to measure in closed-form the goodness of fit between a Gaussian mixture and its polynomial exponential density approximation. This goodness-of-fit divergence is a generalization of the Hyvärinen divergence used to estimate models with computationally intractable normalizers. It allows us to perform model selection by choosing the orders of the polynomial exponential densities used to approximate the mixtures. We demonstrate experimentally that our heuristic to approximate the Jeffreys divergence improves by several orders of magnitude the computational time of stochastic Monte Carlo estimations while approximating reasonably well the Jeffreys divergence, specially when the mixtures have a very small number of modes. Besides, our mixture-to-exponential family conversion techniques may prove useful in other settings.