Understanding the Failure of Batch Normalization for Transformers in NLP
This solves a performance bottleneck for NLP practitioners using Transformers by improving normalization methods, though it is incremental as it builds on existing BN techniques.
The paper investigates why Batch Normalization (BN) underperforms Layer Normalization (LN) in NLP tasks with Transformers, identifying training-inference inconsistency as the key cause and proposing Regularized BN (RBN) to address it, which outperforms or matches LN in 17 out of 20 experimental settings.
Batch Normalization (BN) is a core and prevalent technique in accelerating the training of deep neural networks and improving the generalization on Computer Vision (CV) tasks. However, it fails to defend its position in Natural Language Processing (NLP), which is dominated by Layer Normalization (LN). In this paper, we are trying to answer why BN usually performs worse than LN in NLP tasks with Transformer models. We find that the inconsistency between training and inference of BN is the leading cause that results in the failure of BN in NLP. We define Training Inference Discrepancy (TID) to quantitatively measure this inconsistency and reveal that TID can indicate BN's performance, supported by extensive experiments, including image classification, neural machine translation, language modeling, sequence labeling, and text classification tasks. We find that BN can obtain much better test performance than LN when TID keeps small through training. To suppress the explosion of TID, we propose Regularized BN (RBN) that adds a simple regularization term to narrow the gap between batch statistics and population statistics of BN. RBN improves the performance of BN consistently and outperforms or is on par with LN on 17 out of 20 settings, involving ten datasets and two common variants of Transformer Our code is available at https://github.com/wjxts/RegularizedBN.