Formal Limitations on the Measurement of Mutual Information
This work addresses a foundational issue in machine learning and statistics, showing that mutual information measurement is fundamentally limited, which is significant for researchers and practitioners in information theory and related fields.
The paper tackles the problem of measuring mutual information from finite data by proving that any distribution-free high-confidence lower bound estimated from N samples cannot exceed O(ln N), revealing inherent statistical limitations.
Measuring mutual information from finite data is difficult. Recent work has considered variational methods maximizing a lower bound. In this paper, we prove that serious statistical limitations are inherent to any method of measuring mutual information. More specifically, we show that any distribution-free high-confidence lower bound on mutual information estimated from N samples cannot be larger than O(ln N ).