Improving Deep Transformer with Depth-Scaled Initialization and Merged Attention
This work addresses efficiency and training stability issues for deep neural networks in NLP, particularly for machine translation tasks, though it is incremental as it builds on existing Transformer architectures.
The paper tackled the problem of poor convergence and high computational overhead in deep Transformer models for machine translation by proposing depth-scaled initialization to ease gradient back-propagation and merged attention to reduce computational cost. Results showed an average BLEU improvement of +1.1 on 12-layer models while matching baseline decoding speed.
The general trend in NLP is towards increasing model capacity and performance via deeper neural networks. However, simply stacking more layers of the popular Transformer architecture for machine translation results in poor convergence and high computational overhead. Our empirical analysis suggests that convergence is poor due to gradient vanishing caused by the interaction between residual connections and layer normalization. We propose depth-scaled initialization (DS-Init), which decreases parameter variance at the initialization stage, and reduces output variance of residual connections so as to ease gradient back-propagation through normalization layers. To address computational cost, we propose a merged attention sublayer (MAtt) which combines a simplified averagebased self-attention sublayer and the encoderdecoder attention sublayer on the decoder side. Results on WMT and IWSLT translation tasks with five translation directions show that deep Transformers with DS-Init and MAtt can substantially outperform their base counterpart in terms of BLEU (+1.1 BLEU on average for 12-layer models), while matching the decoding speed of the baseline model thanks to the efficiency improvements of MAtt.