Extreme Entropy Machines: Robust information theoretic classification
This work addresses classification problems for machine learning practitioners by offering a robust, information-theoretic alternative to existing methods, though it appears incremental as it builds on known linear classifiers with an entropic perspective.
The authors tackled classification by proposing Extreme Entropy Machines (EEM), a model based on quadratic Renyi's entropy and Cauchy-Schwarz Divergence, which achieves robustness competitive with state-of-the-art methods like Support Vector Machines and Extreme Learning Machines on datasets ranging from small UCI sets to large, highly unbalanced ones with up to 100:1 class ratios.
Most of the existing classification methods are aimed at minimization of empirical risk (through some simple point-based error measured with loss function) with added regularization. We propose to approach this problem in a more information theoretic way by investigating applicability of entropy measures as a classification model objective function. We focus on quadratic Renyi's entropy and connected Cauchy-Schwarz Divergence which leads to the construction of Extreme Entropy Machines (EEM). The main contribution of this paper is proposing a model based on the information theoretic concepts which on the one hand shows new, entropic perspective on known linear classifiers and on the other leads to a construction of very robust method competetitive with the state of the art non-information theoretic ones (including Support Vector Machines and Extreme Learning Machines). Evaluation on numerous problems spanning from small, simple ones from UCI repository to the large (hundreads of thousands of samples) extremely unbalanced (up to 100:1 classes' ratios) datasets shows wide applicability of the EEM in real life problems and that it scales well.