Is Encoder-Decoder Redundant for Neural Machine Translation?
This work challenges a fundamental assumption in machine translation, potentially simplifying model design for researchers and practitioners, though it appears incremental as it builds on existing language modeling capabilities.
The paper investigates whether the encoder-decoder architecture is necessary for neural machine translation by proposing an alternative approach that concatenates source and target sentences and trains a language model. The results show that this method performs on par with the baseline encoder-decoder Transformer across bilingual, monolingual-augmented, and multilingual translation tasks.
Encoder-decoder architecture is widely adopted for sequence-to-sequence modeling tasks. For machine translation, despite the evolution from long short-term memory networks to Transformer networks, plus the introduction and development of attention mechanism, encoder-decoder is still the de facto neural network architecture for state-of-the-art models. While the motivation for decoding information from some hidden space is straightforward, the strict separation of the encoding and decoding steps into an encoder and a decoder in the model architecture is not necessarily a must. Compared to the task of autoregressive language modeling in the target language, machine translation simply has an additional source sentence as context. Given the fact that neural language models nowadays can already handle rather long contexts in the target language, it is natural to ask whether simply concatenating the source and target sentences and training a language model to do translation would work. In this work, we investigate the aforementioned concept for machine translation. Specifically, we experiment with bilingual translation, translation with additional target monolingual data, and multilingual translation. In all cases, this alternative approach performs on par with the baseline encoder-decoder Transformer, suggesting that an encoder-decoder architecture might be redundant for neural machine translation.