Towards Inducing Long-Context Abilities in Multilingual Neural Machine Translation Models
This addresses the problem of handling long-range dependencies and extended context in multilingual neural machine translation, which is incremental as it adapts existing methods to new positional embeddings.
The paper tackled the challenge of transitioning pre-trained neural machine translation models from absolute Sinusoidal Positional Embeddings to Relative Positional Embeddings like RoPE and ALiBi without performance loss, showing that parameter-efficient fine-tuning with minimal high-quality data achieves competitive sentence-level translation quality and superior document-level performance with RoPE.
Neural Machine Translation (NMT) models have traditionally used Sinusoidal Positional Embeddings (PEs), which often struggle to capture long-range dependencies and are inefficient for handling extended context or document-level translation tasks. This work addresses the challenge of transitioning pre-trained NMT models from absolute Sinusoidal PEs to Relative PEs, such as RoPE and ALiBi, without compromising performance. We demonstrate that parameter-efficient fine-tuning, using only a small amount of high-quality data, can successfully facilitate this transition. Experimental results indicate that switching from Sinusoidal to Relative PEs results in competitive translation quality on sentence-level evaluation benchmarks. Additionally, models trained with RoPE consistently outperform those using ALiBi and Sinusoidal PEs on document-level benchmarks across both string-based metrics and qualitative evaluations. Moreover, we find that a small amount of long-context data in a few languages is sufficient for cross-lingual length generalization, thereby inducing long-context capabilities.