Leveraging Closed-Access Multilingual Embedding for Automatic Sentence Alignment in Low Resource Languages
This work addresses the challenge of creating reliable parallel data for machine translation in low-resource languages, which is incremental as it builds on existing embedding methods.
The paper tackles the problem of obtaining high-quality parallel sentence alignment for low-resource languages by leveraging a closed-access multilingual embedding, achieving F1 scores of 94.96 and 54.83 on FLORES and MAFAND-MT datasets, and improving translation models by over 5 BLEU scores compared to LASER.
The importance of qualitative parallel data in machine translation has long been determined but it has always been very difficult to obtain such in sufficient quantity for the majority of world languages, mainly because of the associated cost and also the lack of accessibility to these languages. Despite the potential for obtaining parallel datasets from online articles using automatic approaches, forensic investigations have found a lot of quality-related issues such as misalignment, and wrong language codes. In this work, we present a simple but qualitative parallel sentence aligner that carefully leveraged the closed-access Cohere multilingual embedding, a solution that ranked second in the just concluded #CoHereAIHack 2023 Challenge (see https://ai6lagos.devpost.com). The proposed approach achieved $94.96$ and $54.83$ f1 scores on FLORES and MAFAND-MT, compared to $3.64$ and $0.64$ of LASER respectively. Our method also achieved an improvement of more than 5 BLEU scores over LASER, when the resulting datasets were used with MAFAND-MT dataset to train translation models. Our code and data are available for research purposes here (https://github.com/abumafrim/Cohere-Align).