Consistent Batch Normalization for Weighted Loss in Imbalanced-Data Environment
This addresses a practical issue in machine learning for imbalanced data classification, but it is incremental as it modifies an existing technique.
The paper tackles the size-inconsistency problem when combining weighted loss functions with batch normalization in imbalanced datasets, proposing weighted batch normalization (WBN) to correct this mismatch and validating it through numerical experiments.
In this study, classification problems based on feedforward neural networks in a data-imbalanced environment are considered. Learning from an imbalanced dataset is one of the most important practical problems in the field of machine learning. A weighted loss function (WLF) based on a cost-sensitive approach is a well-known and effective method for imbalanced datasets. A combination of WLF and batch normalization (BN) is considered in this study. BN is considered as a powerful standard technique in the recent developments in deep learning. A simple combination of both methods leads to a size-inconsistency problem due to a mismatch between the interpretations of the effective size of the dataset in both methods. A simple modification to BN, called weighted BN (WBN), is proposed to correct the size mismatch. The idea of WBN is simple and natural. The proposed method in a data-imbalanced environment is validated using numerical experiments.