Adaptive Batch Normalization for Training Data with Heterogeneous Features
This addresses the problem of inefficient normalization in deep learning for datasets with varying feature homogeneity, offering a pre-training method to improve performance and stability, but it is incremental as it builds on existing BN techniques.
The paper tackles the problem of Batch Normalization (BN) being redundant or degrading for homogeneous datasets by proposing an early-stage feasibility assessment method that classifies training data batches based on feature heterogeneity to decide if normalization is needed. The result shows better performance, mostly in small batch sizes, on MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100 datasets, with increased network stability.
Batch Normalization (BN) is an important preprocessing step to many deep learning applications. Since it is a data-dependent process, for some homogeneous datasets it is a redundant or even a performance-degrading process. In this paper, we propose an early-stage feasibility assessment method for estimating the benefits of applying BN on the given data batches. The proposed method uses a novel threshold-based approach to classify the training data batches into two sets according to their need for normalization. The need for normalization is decided based on the feature heterogeneity of the considered batch. The proposed approach is a pre-training processing, which implies no training overhead. The evaluation results show that the proposed approach achieves better performance mostly in small batch sizes than the traditional BN using MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100 datasets. Additionally, the network stability is increased by reducing the occurrence of internal variable transformation.