On the Sub-Layer Functionalities of Transformer Decoder
This work addresses the understudied decoder in Transformer-based NMT, offering insights that enable more efficient models for translation tasks.
The authors tackled the problem of understanding Transformer decoder functionalities in neural machine translation, revealing that the residual feed-forward module can be removed with minimal performance loss, leading to significant reductions in computation and parameters and boosting training and inference speed.
There have been significant efforts to interpret the encoder of Transformer-based encoder-decoder architectures for neural machine translation (NMT); meanwhile, the decoder remains largely unexamined despite its critical role. During translation, the decoder must predict output tokens by considering both the source-language text from the encoder and the target-language prefix produced in previous steps. In this work, we study how Transformer-based decoders leverage information from the source and target languages -- developing a universal probe task to assess how information is propagated through each module of each decoder layer. We perform extensive experiments on three major translation datasets (WMT En-De, En-Fr, and En-Zh). Our analysis provides insight on when and where decoders leverage different sources. Based on these insights, we demonstrate that the residual feed-forward module in each Transformer decoder layer can be dropped with minimal loss of performance -- a significant reduction in computation and number of parameters, and consequently a significant boost to both training and inference speed.