Depth Growing for Neural Machine Translation
This addresses the challenge of building deeper NMT models for better translation performance, which is incremental as it builds on existing Transformer architectures.
The paper tackled the problem of increasing network depth in neural machine translation (NMT) models to improve translation quality, proposing a two-stage approach that achieved significant improvements over strong Transformer baselines on WMT14 English→German and English→French tasks.
While very deep neural networks have shown effectiveness for computer vision and text classification applications, how to increase the network depth of neural machine translation (NMT) models for better translation quality remains a challenging problem. Directly stacking more blocks to the NMT model results in no improvement and even reduces performance. In this work, we propose an effective two-stage approach with three specially designed components to construct deeper NMT models, which result in significant improvements over the strong Transformer baselines on WMT$14$ English$\to$German and English$\to$French translation tasks\footnote{Our code is available at \url{https://github.com/apeterswu/Depth_Growing_NMT}}.