A Reverse Jensen Inequality Result with Application to Mutual Information Estimation
This work provides a new mathematical tool for information theory and machine learning, with potential applications in mutual information estimation, though it appears incremental as it builds on the well-known Jensen inequality.
The authors tackled the problem of reversing the Jensen inequality under minimal constraints and proper scaling, resulting in a new variational estimator for mutual information that demonstrates superior training behavior compared to current estimators.
The Jensen inequality is a widely used tool in a multitude of fields, such as for example information theory and machine learning. It can be also used to derive other standard inequalities such as the inequality of arithmetic and geometric means or the Hölder inequality. In a probabilistic setting, the Jensen inequality describes the relationship between a convex function and the expected value. In this work, we want to look at the probabilistic setting from the reverse direction of the inequality. We show that under minimal constraints and with a proper scaling, the Jensen inequality can be reversed. We believe that the resulting tool can be helpful for many applications and provide a variational estimation of mutual information, where the reverse inequality leads to a new estimator with superior training behavior compared to current estimators.