Shiksha: A Technical Domain focused Translation Dataset and Model for Indian Languages
This addresses the problem of poor translation performance in scientific and technical domains for low-resource Indian languages, though it is incremental as it builds on existing NMT methods with new data.
The authors tackled the lack of scientific and technical translation data for low-resource Indian languages by creating a multilingual parallel corpus of over 2.8 million high-quality translation pairs across 8 languages, and their finetuned models surpassed other publicly available models on in-domain tasks and improved out-of-domain benchmarks by over 2 BLEU on average.
Neural Machine Translation (NMT) models are typically trained on datasets with limited exposure to Scientific, Technical and Educational domains. Translation models thus, in general, struggle with tasks that involve scientific understanding or technical jargon. Their performance is found to be even worse for low-resource Indian languages. Finding a translation dataset that tends to these domains in particular, poses a difficult challenge. In this paper, we address this by creating a multilingual parallel corpus containing more than 2.8 million rows of English-to-Indic and Indic-to-Indic high-quality translation pairs across 8 Indian languages. We achieve this by bitext mining human-translated transcriptions of NPTEL video lectures. We also finetune and evaluate NMT models using this corpus and surpass all other publicly available models at in-domain tasks. We also demonstrate the potential for generalizing to out-of-domain translation tasks by improving the baseline by over 2 BLEU on average for these Indian languages on the Flores+ benchmark. We are pleased to release our model and dataset via this link: https://huggingface.co/SPRINGLab.