Estimators of Entropy and Information via Inference in Probabilistic Models
This addresses the problem of accurate information estimation in high-dimensional settings for statisticians and machine learning practitioners, offering a method that is incremental/hybrid in nature.
The paper tackles the challenge of estimating information-theoretic quantities like entropy and mutual information in high dimensions by introducing EEVI estimators, which provide upper and lower bounds and demonstrate scalability and efficacy in medical applications, such as ranking medical tests for liver disorders and optimizing blood glucose measurement times for diabetic patients.
Estimating information-theoretic quantities such as entropy and mutual information is central to many problems in statistics and machine learning, but challenging in high dimensions. This paper presents estimators of entropy via inference (EEVI), which deliver upper and lower bounds on many information quantities for arbitrary variables in a probabilistic generative model. These estimators use importance sampling with proposal distribution families that include amortized variational inference and sequential Monte Carlo, which can be tailored to the target model and used to squeeze true information values with high accuracy. We present several theoretical properties of EEVI and demonstrate scalability and efficacy on two problems from the medical domain: (i) in an expert system for diagnosing liver disorders, we rank medical tests according to how informative they are about latent diseases, given a pattern of observed symptoms and patient attributes; and (ii) in a differential equation model of carbohydrate metabolism, we find optimal times to take blood glucose measurements that maximize information about a diabetic patient's insulin sensitivity, given their meal and medication schedule.