Towards Competitive Classifiers for Unbalanced Classification Problems: A Study on the Performance Scores
This work addresses the theoretical gap in evaluating classifiers for imbalanced data, which is crucial for machine learning practitioners dealing with skewed datasets.
The paper tackles the problem of evaluating classifiers in class-imbalanced scenarios by analyzing performance scores, concluding that unweighted Hölder means with exponent p ≤ 1 are adequate for assessing competitiveness. It also reviews existing solutions and provides bounds to ensure classifiers outperform random ones in every class.
Although a great methodological effort has been invested in proposing competitive solutions to the class-imbalance problem, little effort has been made in pursuing a theoretical understanding of this matter. In order to shed some light on this topic, we perform, through a novel framework, an exhaustive analysis of the adequateness of the most commonly used performance scores to assess this complex scenario. We conclude that using unweighted Hölder means with exponent $p \leq 1$ to average the recalls of all the classes produces adequate scores which are capable of determining whether a classifier is competitive. Then, we review the major solutions presented in the class-imbalance literature. Since any learning task can be defined as an optimisation problem where a loss function, usually connected to a particular score, is minimised, our goal, here, is to find whether the learning tasks found in the literature are also oriented to maximise the previously detected adequate scores. We conclude that they usually maximise the unweighted Hölder mean with $p = 1$ (a-mean). Finally, we provide bounds on the values of the studied performance scores which guarantee a classifier with a higher recall than the random classifier in each and every class.