DeepNet: Scaling Transformers to 1,000 Layers
This addresses the challenge of scaling deep neural networks for researchers and practitioners in NLP, enabling more efficient and powerful models, though it is incremental as it builds on existing Transformer architectures.
The paper tackles the problem of training extremely deep Transformers by introducing DeepNorm, a method that stabilizes scaling up to 1,000 layers, resulting in a 200-layer model with 3.2B parameters outperforming a 48-layer state-of-the-art model by 5 BLEU points on a multilingual translation benchmark.
In this paper, we propose a simple yet effective method to stabilize extremely deep Transformers. Specifically, we introduce a new normalization function (DeepNorm) to modify the residual connection in Transformer, accompanying with theoretically derived initialization. In-depth theoretical analysis shows that model updates can be bounded in a stable way. The proposed method combines the best of two worlds, i.e., good performance of Post-LN and stable training of Pre-LN, making DeepNorm a preferred alternative. We successfully scale Transformers up to 1,000 layers (i.e., 2,500 attention and feed-forward network sublayers) without difficulty, which is one order of magnitude deeper than previous deep Transformers. Remarkably, on a multilingual benchmark with 7,482 translation directions, our 200-layer model with 3.2B parameters significantly outperforms the 48-layer state-of-the-art model with 12B parameters by 5 BLEU points, which indicates a promising scaling direction.