Better Multi-class Probability Estimates for Small Data Sets
This work addresses the limitation of existing calibration methods that are ineffective for small datasets and binary-only tasks, offering a solution for applications requiring accurate multi-class probability estimates.
The paper tackles the problem of poor probability calibration for multi-class classification on small datasets, demonstrating that applying the Data Generation and Grouping algorithm reduces calibration error with acceptable computational cost.
Many classification applications require accurate probability estimates in addition to good class separation but often classifiers are designed focusing only on the latter. Calibration is the process of improving probability estimates by post-processing but commonly used calibration algorithms work poorly on small data sets and assume the classification task to be binary. Both of these restrictions limit their real-world applicability. Previously introduced Data Generation and Grouping algorithm alleviates the problem posed by small data sets and in this article, we will demonstrate that its application to multi-class problems is also possible which solves the other limitation. Our experiments show that calibration error can be decreased using the proposed approach and the additional computational cost is acceptable.