Balancing the Scales: A Theoretical and Algorithmic Framework for Learning from Imbalanced Data
It addresses the problem of class imbalance in machine learning, particularly for multi-class long-tailed distributions, with a focus on theoretical foundations, though it is incremental in building upon existing methods.
The paper tackles the challenge of learning from imbalanced data by introducing a theoretical framework and a new class-imbalanced margin loss function, proving its strong H-consistency and achieving improved empirical performance over baselines.
Class imbalance remains a major challenge in machine learning, especially in multi-class problems with long-tailed distributions. Existing methods, such as data resampling, cost-sensitive techniques, and logistic loss modifications, though popular and often effective, lack solid theoretical foundations. As an example, we demonstrate that cost-sensitive methods are not Bayes-consistent. This paper introduces a novel theoretical framework for analyzing generalization in imbalanced classification. We then propose a new class-imbalanced margin loss function for both binary and multi-class settings, prove its strong $H$-consistency, and derive corresponding learning guarantees based on empirical loss and a new notion of class-sensitive Rademacher complexity. Leveraging these theoretical results, we devise novel and general learning algorithms, IMMAX (Imbalanced Margin Maximization), which incorporate confidence margins and are applicable to various hypothesis sets. While our focus is theoretical, we also present extensive empirical results demonstrating the effectiveness of our algorithms compared to existing baselines.