Cross-Iteration Batch Normalization
This addresses a known bottleneck in deep learning for practitioners using small batch sizes, offering an incremental improvement over existing normalization techniques.
The paper tackles the problem of Batch Normalization's reduced effectiveness with small mini-batch sizes by proposing Cross-Iteration Batch Normalization (CBN), which uses examples from multiple recent iterations to improve estimation quality, outperforming original batch normalization and a direct calculation method in object detection and image classification tasks.
A well-known issue of Batch Normalization is its significantly reduced effectiveness in the case of small mini-batch sizes. When a mini-batch contains few examples, the statistics upon which the normalization is defined cannot be reliably estimated from it during a training iteration. To address this problem, we present Cross-Iteration Batch Normalization (CBN), in which examples from multiple recent iterations are jointly utilized to enhance estimation quality. A challenge of computing statistics over multiple iterations is that the network activations from different iterations are not comparable to each other due to changes in network weights. We thus compensate for the network weight changes via a proposed technique based on Taylor polynomials, so that the statistics can be accurately estimated and batch normalization can be effectively applied. On object detection and image classification with small mini-batch sizes, CBN is found to outperform the original batch normalization and a direct calculation of statistics over previous iterations without the proposed compensation technique. Code is available at https://github.com/Howal/Cross-iterationBatchNorm .