Optimal Corpus Aware Training for Neural Machine Translation
This work addresses a specific bottleneck in neural machine translation by making corpus-aware training more efficient and robust, offering a lightweight fine-tuning method that is less sensitive to hyperparameters.
The paper tackles the inefficiency and error-proneness of pre-defining high-quality data in Corpus Aware Training (CAT) for neural machine translation by proposing Optimal Corpus Aware Training (OCAT), which fine-tunes a CAT pre-trained model by tuning only a small set of corpus-related parameters, resulting in improvements of +3.6 and +1.8 chrF on WMT23 English to Chinese and English to German tasks, respectively.
Corpus Aware Training (CAT) leverages valuable corpus metadata during training by injecting corpus information into each training example, and has been found effective in the literature, commonly known as the "tagging" approach. Models trained with CAT inherently learn the quality, domain and nuance between corpora directly from data, and can easily switch to different inference behavior. To achieve the best evaluation, CAT models pre-define a group of high quality data before training starts which can be error-prone and inefficient. In this work, we propose Optimal Corpus Aware Training (OCAT), which fine-tunes a CAT pre-trained model by freezing most of the model parameters and only tuning small set of corpus-related parameters. We show that OCAT is lightweight, resilient to overfitting, and effective in boosting model accuracy. We use WMT23 English to Chinese and English to German translation tasks as our test ground and show +3.6 and +1.8 chrF improvement, respectively, over vanilla training. Furthermore, our approach is on-par or slightly better than other state-of-the-art fine-tuning techniques while being less sensitive to hyperparameter settings.