Vocabulary Adaptation for Distant Domain Adaptation in Neural Machine Translation
This addresses the challenge of adapting neural machine translation models to distant domains (e.g., from movie subtitles to research papers) for practitioners, though it is incremental as it builds on fine-tuning.
The paper tackles the problem of vocabulary mismatches in distant domain adaptation for neural machine translation by proposing vocabulary adaptation, a method that adapts embedding layers to the target domain, resulting in improvements of 3.86 and 3.28 BLEU points in En-Ja and De-En translation, respectively.
Neural network methods exhibit strong performance only in a few resource-rich domains. Practitioners, therefore, employ domain adaptation from resource-rich domains that are, in most cases, distant from the target domain. Domain adaptation between distant domains (e.g., movie subtitles and research papers), however, cannot be performed effectively due to mismatches in vocabulary; it will encounter many domain-specific words (e.g., "angstrom") and words whose meanings shift across domains(e.g., "conductor"). In this study, aiming to solve these vocabulary mismatches in domain adaptation for neural machine translation (NMT), we propose vocabulary adaptation, a simple method for effective fine-tuning that adapts embedding layers in a given pre-trained NMT model to the target domain. Prior to fine-tuning, our method replaces the embedding layers of the NMT model by projecting general word embeddings induced from monolingual data in a target domain onto a source-domain embedding space. Experimental results indicate that our method improves the performance of conventional fine-tuning by 3.86 and 3.28 BLEU points in En-Ja and De-En translation, respectively.