Amended Cross Entropy Cost: Framework For Explicit Diversity Encouragement
This work addresses the need for explicit diversity control in classifier ensembles, offering a method analogous to Negative Correlation Learning for regression, but it appears incremental as it adapts existing cost function principles.
The authors tackled the problem of training multiple classifiers with controlled diversity by introducing the Amended Cross Entropy (ACE) cost function, which yields better ensemble results than vanilla methods, as demonstrated through empirical results.
Cross Entropy (CE) has an important role in machine learning and, in particular, in neural networks. It is commonly used in neural networks as the cost between the known distribution of the label and the Softmax/Sigmoid output. In this paper we present a new cost function called the Amended Cross Entropy (ACE). Its novelty lies in its affording the capability to train multiple classifiers while explicitly controlling the diversity between them. We derived the new cost by mathematical analysis and "reverse engineering" of the way we wish the gradients to behave, and produced a tailor-made, elegant and intuitive cost function to achieve the desired result. This process is similar to the way that CE cost is picked as a cost function for the Softmax/Sigmoid classifiers for obtaining linear derivatives. By choosing the optimal diversity factor we produce an ensemble which yields better results than the vanilla one. We demonstrate two potential usages of this outcome, and present empirical results. Our method works for classification problems analogously to Negative Correlation Learning (NCL) for regression problems.