Guaranteed bounds on the Kullback-Leibler divergence of univariate mixtures using piecewise log-sum-exp inequalities
This provides a practical tool for signal processing tasks where accurate divergence estimation is needed, though it is incremental as it builds on existing bounding techniques.
The paper tackles the problem of computing Kullback-Leibler divergence for mixture models, which lacks a closed-form solution, by introducing a fast method to derive guaranteed lower and upper bounds, demonstrating its effectiveness on univariate exponential, Gaussian, Rayleigh, and Gamma mixtures.
Information-theoretic measures such as the entropy, cross-entropy and the Kullback-Leibler divergence between two mixture models is a core primitive in many signal processing tasks. Since the Kullback-Leibler divergence of mixtures provably does not admit a closed-form formula, it is in practice either estimated using costly Monte-Carlo stochastic integration, approximated, or bounded using various techniques. We present a fast and generic method that builds algorithmically closed-form lower and upper bounds on the entropy, the cross-entropy and the Kullback-Leibler divergence of mixtures. We illustrate the versatile method by reporting on our experiments for approximating the Kullback-Leibler divergence between univariate exponential mixtures, Gaussian mixtures, Rayleigh mixtures, and Gamma mixtures.