Ensembling of Distilled Models from Multi-task Teachers for Constrained Resource Language Pairs
This work addresses translation challenges for low-resource languages, but it is incremental as it builds on existing methods like ensembling and distillation.
The paper tackled machine translation for low-resource language pairs by training multilingual and bilingual models with data augmentation and knowledge distillation, achieving relative BLEU gains of around 70% for English-Hausa and 25% for Bengali-Hindi and Xhosa-Zulu compared to baselines.
This paper describes our submission to the constrained track of WMT21 shared news translation task. We focus on the three relatively low resource language pairs Bengali to and from Hindi, English to and from Hausa, and Xhosa to and from Zulu. To overcome the limitation of relatively low parallel data we train a multilingual model using a multitask objective employing both parallel and monolingual data. In addition, we augment the data using back translation. We also train a bilingual model incorporating back translation and knowledge distillation then combine the two models using sequence-to-sequence mapping. We see around 70% relative gain in BLEU point for English to and from Hausa, and around 25% relative improvements for both Bengali to and from Hindi, and Xhosa to and from Zulu compared to bilingual baselines.