Theoretical Error Analysis of Entropy Approximation for Gaussian Mixtures
This provides theoretical guarantees for entropy approximation in high-dimensional problems, such as uncertainty estimation in neural networks, but is incremental as it builds on existing approximation methods.
The paper tackles the problem of approximating entropy for Gaussian mixtures, which lacks analytical solutions, by theoretically analyzing the error of a simple approximation method. It shows that the error converges to zero as Gaussian components become well-separated, with this condition more likely in high-dimensional spaces like neural networks.
Gaussian mixture distributions are commonly employed to represent general probability distributions. Despite the importance of using Gaussian mixtures for uncertainty estimation, the entropy of a Gaussian mixture cannot be calculated analytically. In this paper, we study the approximate entropy represented as the sum of the entropies of unimodal Gaussian distributions with mixing coefficients. This approximation is easy to calculate analytically regardless of dimension, but there is a lack of theoretical guarantees. We theoretically analyze the approximation error between the true and the approximate entropy to reveal when this approximation works effectively. This error is essentially controlled by how far apart each Gaussian component of the Gaussian mixture is. To measure such separation, we introduce the ratios of the distances between the means to the sum of the variances of each Gaussian component of the Gaussian mixture, and we reveal that the error converges to zero as the ratios tend to infinity. In addition, the probabilistic estimate indicates that this convergence situation is more likely to occur in higher-dimensional spaces. Therefore, our results provide a guarantee that this approximation works well for high-dimensional problems, such as neural networks that involve a large number of parameters.