A Theoretical Analysis of the Learning Dynamics under Class Imbalance
It provides theoretical insights into a common problem in machine learning, though it is incremental as it builds on existing solutions like oversampling.
The paper analyzes how class imbalance affects learning dynamics, showing that gradient-based optimizers lead to sub-optimal trajectories for minority and majority classes, with slowdowns tied to the imbalance ratio, and finds that per-class gradient normalization can address issues in GD but is less effective in SGD due to higher directional noise for minority classes.
Data imbalance is a common problem in machine learning that can have a critical effect on the performance of a model. Various solutions exist but their impact on the convergence of the learning dynamics is not understood. Here, we elucidate the significant negative impact of data imbalance on learning, showing that the learning curves for minority and majority classes follow sub-optimal trajectories when training with a gradient-based optimizer. This slowdown is related to the imbalance ratio and can be traced back to a competition between the optimization of different classes. Our main contribution is the analysis of the convergence of full-batch (GD) and stochastic gradient descent (SGD), and of variants that renormalize the contribution of each per-class gradient. We find that GD is not guaranteed to decrease the loss for each class but that this problem can be addressed by performing a per-class normalization of the gradient. With SGD, class imbalance has an additional effect on the direction of the gradients: the minority class suffers from a higher directional noise, which reduces the effectiveness of the per-class gradient normalization. Our findings not only allow us to understand the potential and limitations of strategies involving the per-class gradients, but also the reason for the effectiveness of previously used solutions for class imbalance such as oversampling.