On the combined effect of class imbalance and concept complexity in deep learning
This addresses the problem of assessing deep learning robustness in imbalanced and complex data scenarios for researchers and practitioners, but it is incremental as it extends known challenges from classical to deep learning without major breakthroughs.
The paper investigates whether deep learning systems can handle class imbalance combined with structural concept complexity, class overlap, and data scarcity, which have challenged classical machine learning. Results on artificial and real-world image datasets show these settings remain mostly challenging for deep learning, with deeper architectures helping only in specific cases like structural complexity but not with overlap or data scarcity, and overfitting often negates advantages in real-world domains.
Structural concept complexity, class overlap, and data scarcity are some of the most important factors influencing the performance of classifiers under class imbalance conditions. When these effects were uncovered in the early 2000s, understandably, the classifiers on which they were demonstrated belonged to the classical rather than Deep Learning categories of approaches. As Deep Learning is gaining ground over classical machine learning and is beginning to be used in critical applied settings, it is important to assess systematically how well they respond to the kind of challenges their classical counterparts have struggled with in the past two decades. The purpose of this paper is to study the behavior of deep learning systems in settings that have previously been deemed challenging to classical machine learning systems to find out whether the depth of the systems is an asset in such settings. The results in both artificial and real-world image datasets (MNIST Fashion, CIFAR-10) show that these settings remain mostly challenging for Deep Learning systems and that deeper architectures seem to help with structural concept complexity but not with overlap challenges in simple artificial domains. Data scarcity is not overcome by deeper layers, either. In the real-world image domains, where overfitting is a greater concern than in the artificial domains, the advantage of deeper architectures is less obvious: while it is observed in certain cases, it is quickly cancelled as models get deeper and perform worse than their shallower counterparts.