The Minimum Information Principle for Discriminative Learning
This work provides a novel framework for building discriminative classifiers, which could benefit researchers and practitioners in machine learning, though it appears incremental as it builds upon existing entropy-based methods.
The authors tackled the problem of classification by proposing a new principle based on minimizing mutual information instead of maximizing entropy, showing that this approach outperforms maximum entropy models in various tasks.
Exponential models of distributions are widely used in machine learning for classiffication and modelling. It is well known that they can be interpreted as maximum entropy models under empirical expectation constraints. In this work, we argue that for classiffication tasks, mutual information is a more suitable information theoretic measure to be optimized. We show how the principle of minimum mutual information generalizes that of maximum entropy, and provides a comprehensive framework for building discriminative classiffiers. A game theoretic interpretation of our approach is then given, and several generalization bounds provided. We present iterative algorithms for solving the minimum information problem and its convex dual, and demonstrate their performance on various classiffication tasks. The results show that minimum information classiffiers outperform the corresponding maximum entropy models.