The Principle of Uncertain Maximum Entropy
This addresses a limitation in statistical estimation for researchers and practitioners, but it appears incremental as it builds directly on a classic principle.
The paper tackles the problem that the Principle of Maximum Entropy requires error-free information by relaxing this requirement using a memoryless communication channel framework, resulting in a new principle that provides an upper bound on entropy and shows experimental performance relative to existing methods.
The Principle of Maximum Entropy is a rigorous technique for estimating an unknown distribution given partial information while simultaneously minimizing bias. However, an important requirement for applying the principle is that the available information be provided error-free (Jaynes, 1982). We relax this requirement using a memoryless communication channel as a framework to derive a new, more general principle. We show our new principle provides an upper bound on the entropy of the unknown distribution and the amount of information lost due to the use of a given communications channel is unknown unless the unknown distribution's entropy is also known. Using our new principle we provide a new interpretation of the classic principle and experimentally show its performance relative to the classic principle and some other generally applicable solutions.