A New Lower Bound for Kullback-Leibler Divergence Based on Hammersley-Chapman-Robbins Bound
This work provides a theoretical tool for information theory and statistics, but it is incremental as it builds on existing bounds without broad practical impact.
The paper tackles the problem of deriving a lower bound for Kullback-Leibler divergence using the Hammersley-Chapman-Robbins bound, resulting in a bound that depends only on the expectation and variance of a chosen function, with equality shown for Bernoulli distributions and convergence to the Cramér-Rao bound for close distributions.
In this paper, we derive a useful lower bound for the Kullback-Leibler divergence (KL-divergence) based on the Hammersley-Chapman-Robbins bound (HCRB). The HCRB states that the variance of an estimator is bounded from below by the Chi-square divergence and the expectation value of the estimator. By using the relation between the KL-divergence and the Chi-square divergence, we show that the lower bound for the KL-divergence which only depends on the expectation value and the variance of a function we choose. This lower bound can also be derived from an information geometric approach. Furthermore, we show that the equality holds for the Bernoulli distributions and show that the inequality converges to the Cramér-Rao bound when two distributions are very close. We also describe application examples and examples of numerical calculation.