On the Impact of Class Imbalance on the Learning Dynamics of Deep Neural Networks:An Intuitive Insight
For researchers developing DNN-based methods for imbalanced data, this work provides an intuitive understanding of the learning dynamics, though it is incremental as it confirms known issues without proposing a solution.
This study investigates how class imbalance affects the learning dynamics of deep neural networks, finding that it causes underfitting of minority classes in early epochs and leads to non-generalizable minority representations due to overfitting. The results highlight that DNNs eventually learn minority samples but only to minimize training loss, resulting in poor test performance.
Class imbalance in deep neural networks (DNNs) has witnessed a rapid increase in research attention in recent years. However, the varying accounts of the reasons behind the poor performance of DNN on imbalance data in pertinent literature shows that little is known about how this agelong phenomenon impacts the performance of DNNs. A better understanding of this problem is crucial to developing effective DNN-based imbalance methods. Thus, this study systematically investigates the impact of class imbalance on the learning dynamics of DNN by monitoring the learning pattern of DNN models on both the majority and minority classes of datasets of varying imbalance ratios. Experimental findings shows that as against learning from balanced datasets where DNN learns the classes similarly, class imbalance has severe deteriorating impact on the performance of DNN, driving the model to underfit the minority class samples in the early training epochs while simultaneously learning only the majority class. Although DNN ultimately learns the minority samples, learning in this manner only results in learnt minority representations that are non-generalizable at test phase because they are merely overfitted to keep the overall training loss as low as possible.