Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Users
This addresses the problem of limited transliteration resources for Indian languages, which is crucial for supporting the next billion users, though it is incremental as it builds on existing datasets and methods.
The authors tackled the lack of publicly available transliteration datasets for Indian languages by creating Aksharantar, a dataset with 26 million transliteration pairs for 21 languages, which is 21 times larger than existing datasets and includes first-time availability for 7 languages and 1 language family. They also developed IndicXlit, a multilingual transliteration model that improves accuracy by 15% on the Dakshina test set.
Transliteration is very important in the Indian language context due to the usage of multiple scripts and the widespread use of romanized inputs. However, few training and evaluation sets are publicly available. We introduce Aksharantar, the largest publicly available transliteration dataset for Indian languages created by mining from monolingual and parallel corpora, as well as collecting data from human annotators. The dataset contains 26 million transliteration pairs for 21 Indic languages from 3 language families using 12 scripts. Aksharantar is 21 times larger than existing datasets and is the first publicly available dataset for 7 languages and 1 language family. We also introduce the Aksharantar testset comprising 103k word pairs spanning 19 languages that enables a fine-grained analysis of transliteration models on native origin words, foreign words, frequent words, and rare words. Using the training set, we trained IndicXlit, a multilingual transliteration model that improves accuracy by 15% on the Dakshina test set, and establishes strong baselines on the Aksharantar testset introduced in this work. The models, mining scripts, transliteration guidelines, and datasets are available at https://github.com/AI4Bharat/IndicXlit under open-source licenses. We hope the availability of these large-scale, open resources will spur innovation for Indic language transliteration and downstream applications. We hope the availability of these large-scale, open resources will spur innovation for Indic language transliteration and downstream applications.