Notes on Generalizing the Maximum Entropy Principle to Uncertain Data
This work addresses a limitation in statistical modeling for scenarios with uncertain or incomplete data, though it appears incremental as it builds on existing latent maximum entropy principles.
The paper tackles the problem of applying the maximum entropy principle when empirical feature expectations cannot be computed due to partially observed variables, introducing uncertain maximum entropy with an expectation-maximization solution. It shows that this technique generalizes the principle of maximum entropy and simplifies use with black box classifiers for sparse, large datasets.
The principle of maximum entropy is a broadly applicable technique for computing a distribution with the least amount of information possible constrained to match empirical data, for instance, feature expectations. We seek to generalize this principle to scenarios where the empirical feature expectations cannot be computed because the model variables are only partially observed, which introduces a dependency on the learned model. Generalizing the principle of latent maximum entropy, we introduce uncertain maximum entropy and describe an expectation-maximization based solution to approximately solve these problems. We show that our technique additionally generalizes the principle of maximum entropy. We additionally discuss the use of black box classifiers with our technique, which simplifies the process of utilizing sparse, large data sets.