Languages Transferred Within the Encoder: On Representation Transfer in Zero-Shot Multilingual Translation
This addresses the challenge of improving zero-shot translation for multilingual NLP systems, offering incremental but practical enhancements to existing models.
The paper tackled the problem of zero-shot translation deficiency in multilingual neural machine translation by analyzing representation transfer, finding that encoders transfer source language to target language subspaces rather than language-agnostic states, and proposed methods that substantially enhanced zero-shot translation performance on datasets like Europarl-15, TED-19, and OPUS-100 without sacrificing supervised directions.
Understanding representation transfer in multilingual neural machine translation (MNMT) can reveal the reason for the zero-shot translation deficiency. In this work, we systematically analyze the representational issue of MNMT models. We first introduce the identity pair, translating a sentence to itself, to address the lack of the base measure in multilingual investigations, as the identity pair can reflect the representation of a language within the model. Then, we demonstrate that the encoder transfers the source language to the representational subspace of the target language instead of the language-agnostic state. Thus, the zero-shot translation deficiency arises because the representation of a translation is entangled with other languages and not transferred to the target language effectively. Based on our findings, we propose two methods: 1) low-rank language-specific embedding at the encoder, and 2) language-specific contrastive learning of the representation at the decoder. The experimental results on Europarl-15, TED-19, and OPUS-100 datasets show that our methods substantially enhance the performance of zero-shot translations without sacrifices in supervised directions by improving language transfer capacity, thereby providing practical evidence to support our conclusions. Codes are available at https://github.com/zhiqu22/ZeroTrans.