Measuring Class-Imbalance Sensitivity of Deterministic Performance Evaluation Metrics
This work addresses a gap in understanding metric sensitivity for practitioners dealing with imbalanced data, which is incremental in providing a framework to avoid common mistakes in metric comparison.
The paper tackles the problem of quantifying how sensitive performance evaluation metrics are to class imbalance in machine learning, revealing that metrics exhibit logarithmic sensitivity where higher imbalance ratios lead to lower sensitivity.
The class-imbalance issue is intrinsic to many real-world machine learning tasks, particularly to the rare-event classification problems. Although the impact and treatment of imbalanced data is widely known, the magnitude of a metric's sensitivity to class imbalance has attracted little attention. As a result, often the sensitive metrics are dismissed while their sensitivity may only be marginal. In this paper, we introduce an intuitive evaluation framework that quantifies metrics' sensitivity to the class imbalance. Moreover, we reveal an interesting fact that there is a logarithmic behavior in metrics' sensitivity meaning that the higher imbalance ratios are associated with the lower sensitivity of metrics. Our framework builds an intuitive understanding of the class-imbalance impact on metrics. We believe this can help avoid many common mistakes, specially the less-emphasized and incorrect assumption that all metrics' quantities are comparable under different class-imbalance ratios.