Neural Machine Translation Models Can Learn to be Few-shot Learners
This provides a more efficient alternative to large language models for domain-specific machine translation tasks.
The paper tackles the problem of enabling smaller neural machine translation models to perform in-context learning for domain adaptation, showing that fine-tuning with a specialized objective allows them to outperform state-of-the-art baselines in translation quality and immediate adaptation rate.
The emergent ability of Large Language Models to use a small number of examples to learn to perform in novel domains and tasks, also called in-context learning (ICL). In this work, we show that a much smaller model can be trained to perform ICL by fine-tuning towards a specialized training objective, exemplified on the task of domain adaptation for neural machine translation. With this capacity for ICL, the model can take advantage of relevant few-shot examples to adapt its output towards the domain. We compare the quality of this domain adaptation to traditional supervised techniques and ICL with a 40B-parameter Large Language Model. Our approach allows efficient batch inference on a mix of domains and outperforms state-of-the-art baselines in terms of both translation quality and immediate adaptation rate, i.e. the ability to reproduce a specific term after being shown a single example.